Wrong Alert generated

[hc4]

Problem description

Just got wrong Alert.

Alert definition:

Alert query generated correctly and it finds item:
messages?rangetype=absolute&from=2016-05-18T15:00:20.372Z&to=2016-05-19T15:00:20.372Z&q=* And it correctly finds message at 2016-05-19T03:08:09.780Z
And I assured, that this message was here before Alert generated.

I've checked logs, but there is no errors.
Only thing I've mentioned, is that just at time of Alert there was index cycling procedure.
And the message was exactly from cycled index (message from graylog2_100 and cycling was from graylog2_100 to graylog2_101)

Environment

• Graylog Version: 2.0.1

[dennisoelkers]

At what time did the index cycling take place?

[hc4]

There is logs from graylog:

2016-05-19T17:59:49.711+03:00 INFO  [AbstractRotationStrategy] Deflector index <graylog2_100> should be rotated, Pointing deflector to new index now!
2016-05-19T17:59:49.711+03:00 INFO  [Deflector] Cycling deflector to next index now.
2016-05-19T17:59:49.712+03:00 INFO  [Deflector] Cycling from <graylog2_100> to <graylog2_101>
2016-05-19T17:59:49.712+03:00 INFO  [Deflector] Creating index target <graylog2_101>...
2016-05-19T17:59:49.996+03:00 INFO  [Indices] Created Graylog index template "graylog-internal" in Elasticsearch.
2016-05-19T17:59:50.152+03:00 INFO  [Deflector] Waiting for index allocation of <graylog2_101>
2016-05-19T17:59:50.312+03:00 INFO  [Deflector] Done!
2016-05-19T17:59:50.312+03:00 INFO  [Deflector] Pointing deflector to new target index....
2016-05-19T17:59:50.466+03:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_100.
2016-05-19T17:59:50.466+03:00 INFO  [SystemJobManager] Submitted SystemJob <56016b10-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2016-05-19T17:59:50.466+03:00 INFO  [SystemJobManager] Submitted SystemJob <56019220-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.SetIndexReadOnlyJob]
2016-05-19T17:59:50.466+03:00 INFO  [SystemJobManager] Submitted SystemJob <56019221-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2016-05-19T17:59:50.466+03:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_101.
2016-05-19T17:59:50.466+03:00 INFO  [Deflector] Done!
2016-05-19T17:59:50.472+03:00 INFO  [MongoIndexRangeService] Calculated range of [graylog2_101] in [1ms].
2016-05-19T17:59:50.473+03:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_101.
2016-05-19T17:59:50.473+03:00 INFO  [SystemJobManager] SystemJob <56019221-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 6ms.
2016-05-19T18:00:19.751+03:00 INFO  [MongoIndexRangeService] Calculated range of [graylog2_100] in [29281ms].
2016-05-19T18:00:19.752+03:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_100.
2016-05-19T18:00:19.752+03:00 INFO  [SystemJobManager] SystemJob <56016b10-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 29286ms.
2016-05-19T18:00:20.466+03:00 INFO  [SetIndexReadOnlyJob] Flushing old index <graylog2_100>.
2016-05-19T18:00:25.858+03:00 INFO  [SetIndexReadOnlyJob] Setting old index <graylog2_100> to read-only.
2016-05-19T18:00:25.938+03:00 INFO  [SystemJobManager] Submitted SystemJob <6b262b20-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2016-05-19T18:00:25.939+03:00 INFO  [SystemJobManager] SystemJob <56019220-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.SetIndexReadOnlyJob] finished in 5472ms.
2016-05-19T18:00:25.939+03:00 INFO  [OptimizeIndexJob] Optimizing index <graylog2_100>.

And Alert message Date: 2016-05-19T15:00:20.372Z (or 2016-05-19T18:00:20.372+03:00 in local time)

[dennisoelkers]

So the situation is that an alert was generated, because to Graylog it seemed as if that specific message was not there for the last day. It was contained in an index though, that was just in the middle of being rotated due to index rotation and alert checking happening at the same time. Am I correct?

[hc4]

yep

[dennisoelkers]

Thanks for supplying the valuable information, a fix was created. After it has been reviewed, we will decide in which one of the future versions it will be released.