/graylog2-server

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream alerts no longer firing after upgrade to 2.0 #2169

Closed
henrikjohansen opened this Issue May 2, 2016 · 10 comments

Comments

Assignees

@dennisoelkers dennisoelkers

Labels
Projects
None yet
Milestone
  2.0.1
5 participants
@henrikjohansen @dennisoelkers @jothoma1 @milgner @kroepke
@henrikjohansen

henrikjohansen commented May 2, 2016

Problem description

Stream alerts are no longer firing after upgrading to 2.0 even though the stream receives messages.

The Message count condition :
Alert is triggered when there is more than one message in the last 3 minutes. Grace period: 0 minutes. Including last message in alert notification.

The above worked just fine in 1.3 but apparently no longer works in 2.0 - no alerts have been generated since upgrading.

There is a discrepancy between editing the existing message count condition and adding a new one which I cannot explain :

screen shot 2016-05-02 at 11 11 33
@henrikjohansen

This comment has been minimized.

Sign in to view

henrikjohansen commented May 2, 2016

To clarify - adding a new message count condition says "than 1 messages in the last 3 minute and " editing the existing one says "than 1 message in the last 3 messages and" ...

@kroepke kroepke added bug S1 P1 labels May 2, 2016

@kroepke kroepke added this to the 2.0.1 milestone May 2, 2016

@kroepke kroepke assigned kroepke and dennisoelkers and unassigned kroepke May 2, 2016

@dennisoelkers

This comment has been minimized.

Sign in to view
Member

dennisoelkers commented May 2, 2016

Fixed the wording glitch. I am not able to reproduce non-firing stream alerts. Is the stream paused? Do you see new stream alerts being added to the list at the bottom of the "Manage Alerts" page?
@henrikjohansen

This comment has been minimized.

Sign in to view

henrikjohansen commented May 3, 2016

No, streams is active and receiving messages. I just deleted & recreated the alert conditions and now alerts are firing again ... I had to do this for several streams ...
@dennisoelkers

This comment has been minimized.

Sign in to view
Member

dennisoelkers commented May 3, 2016

Do you have a backup of your MongoDB before you recreated the stream alert conditions? It would really help if you could extract one that is not working, so we can investigate what's causing this.
@henrikjohansen

This comment has been minimized.

Sign in to view

henrikjohansen commented May 3, 2016

No - unfortunately not ....
@jothoma1

This comment has been minimized.

Sign in to view

jothoma1 commented May 3, 2016

Same for me after upgrading to 2.0 streams alerts didn't work. No mongodb backup...
i'll try to delete & recreate alert condition
@dennisoelkers

This comment has been minimized.

Sign in to view
Member

dennisoelkers commented May 3, 2016

Do you see anything in your server logs?
@milgner

This comment has been minimized.

Sign in to view

milgner commented May 3, 2016

I'm also experiencing this issue. Is there anything specific you want from my MongoDB?
I already looked at db.streams.find().pretty() and db.alarmcallbackconfigurations.find().pretty() and they both look very reasonable.

dennisoelkers added a commit that referenced this issue May 4, 2016

@dennisoelkers
Make alert conditions resilient against changing number formats. 
In 1.3.x numerical alert condition parameters were stored by MongoDB
(resp. the mongo java driver) in a way, that they are returned as
Doubles to the server, while alert conditions persisted in 2.0.0 return
numerical parameters as Integer. Therefore, 2.0.0 cannot handle alert
conditions stored in 1.3.x, so legacy installations upgraded to 2.0.0
do not check any alert conditions anymore.

This change allows to handle both formats by generally assuming they are
Number/Double types and returning their `intValue()`.

Fixes #2169
Loading status checks…
c56706a

@dennisoelkers dennisoelkers referenced this issue May 4, 2016

Merged

Make alert conditions resilient against changing number formats. #2182

dennisoelkers added a commit that referenced this issue May 4, 2016

@dennisoelkers
Make alert conditions resilient against changing number formats. 
In 1.3.x numerical alert condition parameters were stored by MongoDB
(resp. the mongo java driver) in a way, that they are returned as
Doubles to the server, while alert conditions persisted in 2.0.0 return
numerical parameters as Integer. Therefore, 2.0.0 cannot handle alert
conditions stored in 1.3.x, so legacy installations upgraded to 2.0.0
do not check any alert conditions anymore.

This change allows to handle both formats by generally assuming they are
Number/Double types and returning their `intValue()`.

Fixes #2169
9c6fe6f

joschi added a commit that referenced this issue May 9, 2016

@dennisoelkers @joschi
Make alert conditions resilient against changing number formats. (#2182) 
In 1.3.x numerical alert condition parameters were stored by MongoDB
(resp. the Mongo Java Driver) in a way, that they are returned as
Double  while alert conditions persisted in 2.0.0 return numerical parameters
as Integer. Therefore, 2.0.0 cannot handle alert conditions stored in 1.3.x
and legacy installations upgraded to 2.0.0 do not check any alert conditions
anymore.

This change allows to handle both formats by generally assuming they are
Number/Double types and returning their `intValue()`.

Fixes #2169
Loading status checks…
d164e7f

@dennisoelkers dennisoelkers closed this in ea4aa95 May 9, 2016

@dennisoelkers

This comment has been minimized.

Sign in to view
Member

dennisoelkers commented May 9, 2016

This has been fixed and the fix will be included in 2.0.1. Thanks for helping, everybody!
@milgner

This comment has been minimized.

Sign in to view

milgner commented Jun 10, 2016

Problem still persists after upgrading to 2.0.1. I tried re-creating the alert conditions for the streams but still no luck. Is there any script that needs to be run to get it working again?

I considered that it may be a problem with the notifier, but the "Triggered alerts" section on the stream configuration page also says "This stream has never triggered an alert." even though there are many messages in the stream that should have triggered the alert.

The current configuration is: Alert is triggered when there are more than 3 messages in the last 5 minutes. Grace period: 15 minutes. Including last 3 messages in alert notification.

Should I open a new issue?

@kroepke kroepke added the triaged label Sep 21, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment