Alert on field content?

[henrikjohansen]

I am sure that this has been mentioned somewhere previously but it would be awesome to be able to alert based on field content (string and regex matching) - currently field condition alerts only support metrics & statistics.

Inverting matches should be supported, too.

[lennartkoopmann]

This sounds like the old "forwarders" feature, where you could forward every message that comes into a stream to a given location or a plugin.

Duplicate #425

[henrikjohansen]

Well - if forwarders allow you to alert based on "link up", "link down", "duplicate ip" and strings like that in a specific field then yes .....

[lennartkoopmann]

Forwarders not, but streams and stream rules.

[dennisoelkers]

I think what @henrikjohansen wants is different from stream forwarders. But although henriks requirement is solvable by using the current implementation of stream rules and alert conditions (basically by just filtering for the field content you want and defining an alert condition that is triggered for every incoming message), I think we should implement this feature as a new alert condition type because there are lots of scenarios where it makes sense (and it is trivial to implement).

[timukas]

Any chances to see this feature in upcoming beta?

[dennisoelkers]

Not for the upcoming beta, but probably for a later 0.21.x version.

[dfch]

I also find this feature very useful (and to have more flexible alert checks in general). Is it possible to extend Graylog2 in this area as a plugin? Or is there any other way to program this oneself?

[kroepke]

this needs lots of ui work and as such isn't possible with plugins right now.
postponing until we have a better idea on how to fix this.

[dfch]

thanks for the feedback! I still have a question regarding that: you mention this would present a lot of UI work; I could still find this useful even if it is only available via config file (like the drools engine support). If you could get me a hint on where and how you best see to enhance this then I could start myself and present this for review via pull request

[henrikjohansen]

@kroepke Sorry, I don't get this "UI needs lots of work" thing ... rename the "Field value condition" to "Field metric condition" (because that's what it really is) ... add a "Field string condition" and you're done ?

This should be trivial to implement ?

Extending alert conditions using plugins should be an entirely different issue ;-)

[dfch]

@kroepke Is this still on the milestone:1.1.0 roadmap? Currently nothing seems to have been done for that.
Regarding the comment on "this needs lots of ui work" I would suggest to first implement this in the API/graylog2-server and only later implement this in the UI as well. If this is a viable approach then I could offer to implement this in the server core and submit a PR.

[kroepke]

Currently we have not addressed this because of the UI problems (which stem from the fact that the types of alert conditions are also hardcoded in the interface, and need markup and handling in the web interface project). Unfortunately it's not trivial to implement, but of course not impossible. But rather than adding more and more special case code we'd like to solve the problem "correctly", but for that we need support for UI in plugins. And that's a lot of work.

Let me discuss internally if we can live with having no UI for this feature at this point, @dfch. I'll comment here, or you can contact us directly, too, of course.

[kroepke]

@dfch Just discussed this and we agree that even without a UI this adds enough value and we would be happy to review a PR once you have it in a state where it makes sense to start looking.
Feel free to issue the PR early if you want feedback on implementation details, we can iterate that way, too :)

Thank you for volunteering the work!

[dfch]

@kroepke cool! I just started with a basic field content comparison and am currently testing it. I think I understood how you are handling the alert conditions, though there are two things which are not yet totally clear to me. but for now I think I have enough to continue, otherwise I will ask.
Should I commit to a special feature-branch or how do you want to have that PR?

@henrikjohansen is there something special you had in mind when asking for that feature? I thought about implementing (1) a static field content (string) comparison and (2) an additional regex comparison condition. Plus we would need (3) a condition that allows for comparing more than one field. If you have some other interesting use case, pls let me know.

Ronald

[kroepke]

Just on master should be fine :)
On Apr 9, 2015 5:31 PM, "Ronald Rink" notifications@github.com wrote:

@kroepke https://github.com/kroepke cool! I just started with a basic
field content comparison and am currently testing it. I think I understood
how you are handling the alert conditions, though there are two things
which are not yet totally clear to me. but for now I think I have enough to
continue, otherwise I will ask.
Should I commit to a special feature-branch or how do you want to have
that PR?

@henrikjohansen https://github.com/henrikjohansen is there something
special you had in mind when asking for that feature? I thought about
implementing (1) a static field content (string) comparison and (2) an
additional regex comparison condition. Plus we would need (3) a condition
that allows for comparing more than one field. If you have some other
interesting use case, pls let me know.

Ronald

—
Reply to this email directly or view it on GitHub
#537 (comment)
.

[henrikjohansen]

@dfch I think all our use cases would be covered in what you have outlined. This is probably one of the biggest things that we're missing so I'm really looking forward to having this functionality :)

[lennartkoopmann]

The problem I see with implementing this using the standard alert system is that the alerts are triggered by periodically performing a search against Elasticsearch. This can work with normal string matching but can get crazy expensive for regex searches.

My first comment on this ticket suggested to create a stream that is catching all the messages you want to alert on using a regular expression on the field content. You can then set a message count alert on it and be alerted whenever such a message was received. What use case do I miss here? Too many of those streams and too hard to manage because of poor UI?

[henrikjohansen]

Streams are not 'free' either - they do carry a price tag in terms of maintenance and performance costs.
We might have messages from certain sources that need to be evaluated against 30 or 40 stream ... judging from my experiences with running Graylog this will impact our server nodes a lot.

I have a list of over 100+ strings that we would like (need) to get alerts for (and more to come) - do you really consider maintaining 100+ streams for alerting purposes a viable option? (the streams approach is further complicated by the fact that stream rules are always evaluated using AND resulting in an explosion of streams in our case).

[lennartkoopmann]

Got it. So this will boil down to the question where to put the load: Elasticsearch or graylog-server.

[henrikjohansen]

... and if you can afford the management headache of maintaining 100+ streams for this or if you rather would throw a bit more machine power at the problem (do not underestimate the management cost of a stream based solution).

If you still think that streams are the way to go for solving this then #381 and #382 should be taken into consideration for easing the entire process.