New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remember last rotation time for an index #725

Closed
kroepke opened this Issue Sep 25, 2014 · 10 comments

Comments

Projects
None yet
6 participants
@kroepke
Member

kroepke commented Sep 25, 2014

To avoid having to rotate on startup with time based retention we need to know the time when an index was created and rotated.
Create a utility type and document in each elasticsearch index to store metadata in.
This can also be used for index ranges.

Follow up to #137

@kroepke kroepke self-assigned this Sep 25, 2014

@kroepke kroepke added this to the 0.92 milestone Sep 25, 2014

@joschi joschi modified the milestone: 0.92 Jan 13, 2015

@bernd bernd added this to the 1.2.0 milestone Aug 7, 2015

kroepke added a commit that referenced this issue Aug 25, 2015

use last rotation time initially when locating the anchor used for ro…
…tation

 - fixes missing rotation if the downtime missed more than one period

fixes #725
fixes #829

@kroepke kroepke closed this in 22b8fca Aug 26, 2015

@Ubay

This comment has been minimized.

Ubay commented Sep 16, 2015

After upgrading to version 1.2.0 i'm still facing this issue:

2015-09-16T12:30:21.097+01:00 INFO  [IndexRotationThread] Deflector index <graylog2_142> should be rotated, Pointing deflector to new index now!
2015-09-16T12:30:21.097+01:00 INFO  [Deflector] Cycling deflector to next index now.
2015-09-16T12:30:21.131+01:00 INFO  [Deflector] Cycling from <graylog2_142> to <graylog2_143>
2015-09-16T12:30:21.132+01:00 INFO  [Deflector] Creating index target <graylog2_143>...
2015-09-16T12:30:21.644+01:00 INFO  [Deflector] Waiting for index allocation of <graylog2_143>
2015-09-16T12:30:21.646+01:00 INFO  [Deflector] Done!
2015-09-16T12:30:21.647+01:00 INFO  [Deflector] Pointing deflector to new target index....

After restarting server the deflector is rotated again:

2015-09-16T12:59:18.830+01:00 INFO  [IndexRotationThread] Deflector index <graylog2_143> should be rotated, Pointing deflector to new index now!
2015-09-16T12:59:18.830+01:00 INFO  [Deflector] Cycling deflector to next index now.
2015-09-16T12:59:18.869+01:00 INFO  [Deflector] Cycling from <graylog2_143> to <graylog2_144>
2015-09-16T12:59:18.870+01:00 INFO  [Deflector] Creating index target <graylog2_144>...
2015-09-16T12:59:19.257+01:00 INFO  [Deflector] Waiting for index allocation of <graylog2_144>
2015-09-16T12:59:19.260+01:00 INFO  [Deflector] Done!
2015-09-16T12:59:19.260+01:00 INFO  [Deflector] Pointing deflector to new target index....

Thank you.

@kroepke

This comment has been minimized.

Member

kroepke commented Sep 16, 2015

Could you please provide a debug log for the logger: org.graylog2.periodical.IndexRotationThread ?

@kroepke kroepke reopened this Sep 16, 2015

@joschi

This comment has been minimized.

Contributor

joschi commented Sep 16, 2015

Also, I notice that the log statements you pasted are edited, this makes it really difficult to help.

Those were just display glitches because of the < and > characters around index names. I've added code markup to the previous post to improve readability.

@kroepke

This comment has been minimized.

Member

kroepke commented Sep 16, 2015

Also, please provide the configuration for the rotation settings.

@Ubay

This comment has been minimized.

Ubay commented Sep 16, 2015

The rotation settings are:

rotation_strategy = time
elasticsearch_max_time_per_index = 4w

The debug log is:

2015-09-16T13:39:26.373+01:00 DEBUG [IndexRotationThread] Rotation strategy result: Rotation period P4W elapsed, next rotation at 2015-08-31T00:00:00.000Z
2015-09-16T13:39:26.373+01:00 INFO  [IndexRotationThread] Deflector index <graylog2_149> should be rotated, Pointing deflector to new index now!
2015-09-16T13:39:26.374+01:00 INFO  [Deflector] Cycling deflector to next index now.
2015-09-16T13:39:26.419+01:00 INFO  [Deflector] Cycling from <graylog2_149> to <graylog2_150>
2015-09-16T13:39:26.420+01:00 INFO  [Deflector] Creating index target <graylog2_150>...
2015-09-16T13:39:26.780+01:00 INFO  [Deflector] Waiting for index allocation of <graylog2_150>
2015-09-16T13:39:26.784+01:00 INFO  [Deflector] Done!
2015-09-16T13:39:26.784+01:00 INFO  [Deflector] Pointing deflector to new target index....
2015-09-16T13:39:26.858+01:00 INFO  [SystemJobManager] Submitted SystemJob <f7860971-5c6f-11e5-8d19-0050569e1536> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-09-16T13:39:26.860+01:00 INFO  [SystemJobManager] Submitted SystemJob <f786a5b0-5c6f-11e5-8d19-0050569e1536> [org.graylog2.indexer.SetIndexReadOnlyJob]
2015-09-16T13:39:26.861+01:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_149.
2015-09-16T13:39:26.861+01:00 INFO  [SystemJobManager] Submitted SystemJob <f786ccc0-5c6f-11e5-8d19-0050569e1536> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-09-16T13:39:26.861+01:00 INFO  [Deflector] Done!
2015-09-16T13:39:26.863+01:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_150.
2015-09-16T13:39:26.894+01:00 INFO  [EsIndexRangeService] Calculated range of [graylog2_150] in [27ms].
2015-09-16T13:39:26.897+01:00 INFO  [EsIndexRangeService] Calculated range of [graylog2_149] in [35ms].
2015-09-16T13:39:26.941+01:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_149.
2015-09-16T13:39:26.942+01:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_150.
2015-09-16T13:39:26.947+01:00 INFO  [SystemJobManager] SystemJob <f786ccc0-5c6f-11e5-8d19-0050569e1536> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 84ms.
2015-09-16T13:39:26.942+01:00 INFO  [SystemJobManager] SystemJob <f7860971-5c6f-11e5-8d19-0050569e1536> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 81ms.
2015-09-16T13:39:36.324+01:00 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-09-28T00:00:00.000Z
2015-09-16T13:39:36.324+01:00 DEBUG [IndexRotationThread] Deflector index <graylog2_150> should not be rotated. Not doing anything.

Thank you very much.

@gruselglatz

This comment has been minimized.

gruselglatz commented Nov 16, 2015

I am Facing the same Problem and its very anoying.

Settings:

rotation_strategy = time
elasticsearch_max_time_per_index = 24h

but theelasticsearch_max_size_per_index = 1073741824is setting itself, is this normal?

Debug Log:

 [IndexRotationThread] Deflector index <graylog_101> should be rotated, Pointing deflector to new index now!
2015-11-16_12:06:00.32795 INFO  [Deflector] Cycling deflector to next index now.
2015-11-16_12:06:00.36792 INFO  [Deflector] Cycling from <graylog_101> to <graylog_102>
2015-11-16_12:06:00.36797 INFO  [Deflector] Creating index target <graylog_102>...
2015-11-16_12:06:00.61518 INFO  [Deflector] Waiting for index allocation of <graylog_102>
2015-11-16_12:06:00.61613 INFO  [Deflector] Done!
2015-11-16_12:06:00.61614 INFO  [Deflector] Pointing deflector to new target index....
2015-11-16_12:06:00.66157 INFO  [SystemJobManager] Submitted SystemJob <66efb940-8c5a-11e5-b748-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-16_12:06:00.66169 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_101.
2015-11-16_12:06:00.66190 INFO  [SystemJobManager] Submitted SystemJob <66efe050-8c5a-11e5-b748-005056ad0488> [org.graylog2.indexer.SetIndexReadOnlyJob]
2015-11-16_12:06:00.66942 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_102.
2015-11-16_12:06:00.66967 INFO  [SystemJobManager] Submitted SystemJob <66efe051-8c5a-11e5-b748-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-16_12:06:00.66971 INFO  [Deflector] Done!

thank you

@joschi

This comment has been minimized.

Contributor

joschi commented Nov 17, 2015

@renapu Unfortunately you didn't activate the debug log. You can either start Graylog with the -d or --debug parameter, edit your local log4j.xml file, or activate DEBUG log for org.graylog2.periodical.IndexRotationThread via the Graylog REST API.

@gruselglatz

This comment has been minimized.

gruselglatz commented Nov 17, 2015

here is my debug log:

2015-11-17_11:56:03.63310 DEBUG [IndexRotationThread] Rotation strategy result: Rotation period P1D elapsed, next rotation at 2015-11-17T00:00:00.000Z
2015-11-17_11:56:03.63327 INFO  [IndexRotationThread] Deflector index <graylog_109> should be rotated, Pointing deflector to new index now!
2015-11-17_11:56:03.63336 INFO  [Deflector] Cycling deflector to next index now.
2015-11-17_11:56:03.71224 INFO  [Deflector] Cycling from <graylog_109> to <graylog_110>
2015-11-17_11:56:03.71232 INFO  [Deflector] Creating index target <graylog_110>...
2015-11-17_11:56:04.32571 INFO  [Deflector] Waiting for index allocation of <graylog_110>
2015-11-17_11:56:04.33076 INFO  [Deflector] Done!
2015-11-17_11:56:04.33079 INFO  [Deflector] Pointing deflector to new target index....
2015-11-17_11:56:04.44970 INFO  [SystemJobManager] Submitted SystemJob <2df89b21-8d22-11e5-9410-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-17_11:56:04.45033 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_109.
2015-11-17_11:56:04.45767 INFO  [SystemJobManager] Submitted SystemJob <2dfae512-8d22-11e5-9410-005056ad0488> [org.graylog2.indexer.SetIndexReadOnlyJob]
2015-11-17_11:56:04.46565 INFO  [SystemJobManager] Submitted SystemJob <2dfc1d90-8d22-11e5-9410-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-17_11:56:04.46568 INFO  [Deflector] Done!
2015-11-17_11:56:04.48927 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_110.
2015-11-17_11:56:04.73538 INFO  [MongoIndexRangeService] Calculated range of [graylog_110] in [245ms].
2015-11-17_11:56:04.77822 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_110.
2015-11-17_11:56:04.79046 INFO  [SystemJobManager] SystemJob <2dfc1d90-8d22-11e5-9410-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 318ms.
2015-11-17_11:56:07.18284 INFO  [MongoIndexRangeService] Calculated range of [graylog_109] in [2727ms].
2015-11-17_11:56:07.20974 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_109.
2015-11-17_11:56:07.20977 INFO  [SystemJobManager] SystemJob <2df89b21-8d22-11e5-9410-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 2759ms.
2015-11-17_11:56:13.26365 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:56:13.26368 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:56:23.26627 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:56:23.26636 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:56:33.26308 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:56:33.26318 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:56:34.45026 INFO  [SetIndexReadOnlyJob] Flushing old index <graylog_109>.
2015-11-17_11:56:34.48608 INFO  [SetIndexReadOnlyJob] Setting old index <graylog_109> to read-only.
2015-11-17_11:56:34.51424 INFO  [SystemJobManager] Submitted SystemJob <3fe64c11-8d22-11e5-9410-005056ad0488> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2015-11-17_11:56:34.51433 INFO  [OptimizeIndexJob] Optimizing index <graylog_109>.
2015-11-17_11:56:34.51442 INFO  [SystemJobManager] SystemJob <2dfae512-8d22-11e5-9410-005056ad0488> [org.graylog2.indexer.SetIndexReadOnlyJob] finished in 64ms.
2015-11-17_11:56:43.26867 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:56:43.26953 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:56:53.24680 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:56:53.24683 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:57:03.24650 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:57:03.24660 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:57:13.27139 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:57:13.27147 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:57:23.24969 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:57:23.24978 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:57:33.23609 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:57:33.23614 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:57:43.27427 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:57:43.27435 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:57:53.23857 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:57:53.23872 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.
2015-11-17_11:58:03.23506 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-18T00:00:00.000Z
2015-11-17_11:58:03.23508 DEBUG [IndexRotationThread] Deflector index <graylog_110> should not be rotated. Not doing anything.

@gruselglatz

This comment has been minimized.

gruselglatz commented Nov 26, 2015

The Problem seems to be that the Rotation Date gets reseted see the following logs:

2015-11-26_14:16:33.16688 DEBUG [IndexRotationThread] Rotation strategy result: Rotation period P1D elapsed, next rotation at 2015-11-26T00:00:00.000Z
2015-11-26_14:16:33.16698 INFO  [IndexRotationThread] Deflector index <graylog_133> should be rotated, Pointing deflector to new index now!
2015-11-26_14:16:33.16705 INFO  [Deflector] Cycling deflector to next index now.
2015-11-26_14:16:33.30600 INFO  [Deflector] Cycling from <graylog_133> to <graylog_134>
2015-11-26_14:16:33.30610 INFO  [Deflector] Creating index target <graylog_134>...
2015-11-26_14:16:33.56245 INFO  [Deflector] Waiting for index allocation of <graylog_134>
2015-11-26_14:16:33.56362 INFO  [Deflector] Done!
2015-11-26_14:16:33.56367 INFO  [Deflector] Pointing deflector to new target index....
2015-11-26_14:16:33.58466 INFO  [SystemJobManager] Submitted SystemJob <4bda6bf0-9448-11e5-8c83-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-26_14:16:33.58480 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_133.
2015-11-26_14:16:33.58498 INFO  [SystemJobManager] Submitted SystemJob <4bda9300-9448-11e5-8c83-005056ad0488> [org.graylog2.indexer.SetIndexReadOnlyJob]
2015-11-26_14:16:33.58630 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_134.
2015-11-26_14:16:33.58634 INFO  [SystemJobManager] Submitted SystemJob <4bdaba10-9448-11e5-8c83-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-26_14:16:33.58637 INFO  [Deflector] Done!
2015-11-26_14:16:33.63410 INFO  [MongoIndexRangeService] Calculated range of [graylog_134] in [47ms].
2015-11-26_14:16:33.64238 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_134.
2015-11-26_14:16:33.64340 INFO  [SystemJobManager] SystemJob <4bdaba10-9448-11e5-8c83-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 57ms.
2015-11-26_14:16:33.72285 INFO  [MongoIndexRangeService] Calculated range of [graylog_133] in [133ms].
2015-11-26_14:16:33.72415 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_133.
2015-11-26_14:16:33.72501 INFO  [SystemJobManager] SystemJob <4bda6bf0-9448-11e5-8c83-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 139ms.
2015-11-26_14:16:43.04029 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-27T00:00:00.000Z
2015-11-26_14:16:43.04033 DEBUG [IndexRotationThread] Deflector index <graylog_134> should not be rotated. Not doing anything.
2015-11-26_14:16:53.02359 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-27T00:00:00.000Z
2015-11-26_14:16:53.02364 DEBUG [IndexRotationThread] Deflector index <graylog_134> should not be rotated. Not doing anything.
2015-11-26_14:17:03.05762 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-27T00:00:00.000Z
2015-11-26_14:17:03.05765 DEBUG [IndexRotationThread] Deflector index <graylog_134> should not be rotated. Not doing anything.

after that:

2015-11-26_14:19:46.62541 DEBUG [IndexRotationThread] Rotation strategy result: Rotation period P1D elapsed, next rotation at 2015-11-26T00:00:00.000Z
2015-11-26_14:19:46.62547 INFO  [IndexRotationThread] Deflector index <graylog_134> should be rotated, Pointing deflector to new index now!
2015-11-26_14:19:46.62551 INFO  [Deflector] Cycling deflector to next index now.
2015-11-26_14:19:46.82907 INFO  [Deflector] Cycling from <graylog_134> to <graylog_135>
2015-11-26_14:19:46.82909 INFO  [Deflector] Creating index target <graylog_135>...
2015-11-26_14:19:47.20549 INFO  [Deflector] Waiting for index allocation of <graylog_135>
2015-11-26_14:19:47.20777 INFO  [Deflector] Done!
2015-11-26_14:19:47.20780 INFO  [Deflector] Pointing deflector to new target index....
2015-11-26_14:19:47.23663 INFO  [SystemJobManager] Submitted SystemJob <bf475f30-9448-11e5-bea5-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-26_14:19:47.23667 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_134.
2015-11-26_14:19:47.23704 INFO  [SystemJobManager] Submitted SystemJob <bf478640-9448-11e5-bea5-005056ad0488> [org.graylog2.indexer.SetIndexReadOnlyJob]
2015-11-26_14:19:47.24203 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_135.
2015-11-26_14:19:47.24213 INFO  [SystemJobManager] Submitted SystemJob <bf47ad50-9448-11e5-bea5-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2015-11-26_14:19:47.24216 INFO  [Deflector] Done!
2015-11-26_14:19:47.29833 INFO  [MongoIndexRangeService] Calculated range of [graylog_135] in [56ms].
2015-11-26_14:19:47.30316 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_135.
2015-11-26_14:19:47.30402 INFO  [SystemJobManager] SystemJob <bf47ad50-9448-11e5-bea5-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 66ms.
2015-11-26_14:19:47.36646 INFO  [MongoIndexRangeService] Calculated range of [graylog_134] in [128ms].
2015-11-26_14:19:47.36784 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_134.
2015-11-26_14:19:47.36852 INFO  [SystemJobManager] SystemJob <bf475f30-9448-11e5-bea5-005056ad0488> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 131ms.
2015-11-26_14:19:56.10492 DEBUG [IndexRotationThread] Rotation strategy result: Next rotation at 2015-11-27T00:00:00.000Z
2015-11-26_14:19:56.10497 DEBUG [IndexRotationThread] Deflector index <graylog_135> should not be rotated. Not doing anything.
@macfergus

This comment has been minimized.

macfergus commented Jan 14, 2016

Hi all, we've been debugging the same problem. I think I found the bug; seems like it's just an incorrect config key.

Our setup is graylog 1.3.2, elasticsearch 1.7.4, running on ubuntu 14.04

joschi added a commit that referenced this issue Jan 15, 2016

@joschi joschi closed this in 30cc994 Jan 15, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment