New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong Alert generated #2264

Closed
hc4 opened this Issue May 19, 2016 · 5 comments

Comments

Projects
None yet
4 participants
@hc4
Contributor

hc4 commented May 19, 2016

Problem description

Just got wrong Alert.

Alert definition:
image

Alert query generated correctly and it finds item:
messages?rangetype=absolute&from=2016-05-18T15:00:20.372Z&to=2016-05-19T15:00:20.372Z&q=* And it correctly finds message at 2016-05-19T03:08:09.780Z
And I assured, that this message was here before Alert generated.

I've checked logs, but there is no errors.
Only thing I've mentioned, is that just at time of Alert there was index cycling procedure.
And the message was exactly from cycled index (message from graylog2_100 and cycling was from graylog2_100 to graylog2_101)

Environment

  • Graylog Version: 2.0.1

@hc4 hc4 changed the title from Unexpected Alert generated to Wrong Alert generated May 19, 2016

@dennisoelkers

This comment has been minimized.

Member

dennisoelkers commented May 19, 2016

At what time did the index cycling take place?

@hc4

This comment has been minimized.

Contributor

hc4 commented May 19, 2016

There is logs from graylog:

2016-05-19T17:59:49.711+03:00 INFO  [AbstractRotationStrategy] Deflector index <graylog2_100> should be rotated, Pointing deflector to new index now!
2016-05-19T17:59:49.711+03:00 INFO  [Deflector] Cycling deflector to next index now.
2016-05-19T17:59:49.712+03:00 INFO  [Deflector] Cycling from <graylog2_100> to <graylog2_101>
2016-05-19T17:59:49.712+03:00 INFO  [Deflector] Creating index target <graylog2_101>...
2016-05-19T17:59:49.996+03:00 INFO  [Indices] Created Graylog index template "graylog-internal" in Elasticsearch.
2016-05-19T17:59:50.152+03:00 INFO  [Deflector] Waiting for index allocation of <graylog2_101>
2016-05-19T17:59:50.312+03:00 INFO  [Deflector] Done!
2016-05-19T17:59:50.312+03:00 INFO  [Deflector] Pointing deflector to new target index....
2016-05-19T17:59:50.466+03:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_100.
2016-05-19T17:59:50.466+03:00 INFO  [SystemJobManager] Submitted SystemJob <56016b10-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2016-05-19T17:59:50.466+03:00 INFO  [SystemJobManager] Submitted SystemJob <56019220-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.SetIndexReadOnlyJob]
2016-05-19T17:59:50.466+03:00 INFO  [SystemJobManager] Submitted SystemJob <56019221-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob]
2016-05-19T17:59:50.466+03:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog2_101.
2016-05-19T17:59:50.466+03:00 INFO  [Deflector] Done!
2016-05-19T17:59:50.472+03:00 INFO  [MongoIndexRangeService] Calculated range of [graylog2_101] in [1ms].
2016-05-19T17:59:50.473+03:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_101.
2016-05-19T17:59:50.473+03:00 INFO  [SystemJobManager] SystemJob <56019221-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 6ms.
2016-05-19T18:00:19.751+03:00 INFO  [MongoIndexRangeService] Calculated range of [graylog2_100] in [29281ms].
2016-05-19T18:00:19.752+03:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog2_100.
2016-05-19T18:00:19.752+03:00 INFO  [SystemJobManager] SystemJob <56016b10-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.ranges.CreateNewSingleIndexRangeJob] finished in 29286ms.
2016-05-19T18:00:20.466+03:00 INFO  [SetIndexReadOnlyJob] Flushing old index <graylog2_100>.
2016-05-19T18:00:25.858+03:00 INFO  [SetIndexReadOnlyJob] Setting old index <graylog2_100> to read-only.
2016-05-19T18:00:25.938+03:00 INFO  [SystemJobManager] Submitted SystemJob <6b262b20-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2016-05-19T18:00:25.939+03:00 INFO  [SystemJobManager] SystemJob <56019220-1dd2-11e6-a59f-005056b37533> [org.graylog2.indexer.SetIndexReadOnlyJob] finished in 5472ms.
2016-05-19T18:00:25.939+03:00 INFO  [OptimizeIndexJob] Optimizing index <graylog2_100>.

And Alert message Date: 2016-05-19T15:00:20.372Z (or 2016-05-19T18:00:20.372+03:00 in local time)

@dennisoelkers dennisoelkers self-assigned this May 20, 2016

@joschi joschi added this to the 2.1.0 milestone May 23, 2016

@dennisoelkers

This comment has been minimized.

Member

dennisoelkers commented May 24, 2016

So the situation is that an alert was generated, because to Graylog it seemed as if that specific message was not there for the last day. It was contained in an index though, that was just in the middle of being rotated due to index rotation and alert checking happening at the same time. Am I correct?

@hc4

This comment has been minimized.

Contributor

hc4 commented May 24, 2016

yep

@dennisoelkers

This comment has been minimized.

Member

dennisoelkers commented May 24, 2016

Thanks for supplying the valuable information, a fix was created. After it has been reviewed, we will decide in which one of the future versions it will be released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment