Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test for existing Grok-Pattern during pipeline-processing #5689

Closed
MarsM0nd opened this issue Feb 17, 2019 · 0 comments · Fixed by #5699

Comments

@MarsM0nd
Copy link

commented Feb 17, 2019

The idea is to add either a check if a string is a valid name of a Grok-Pattern, or generally is a Grok-Pattern that can be used, or to allow the grok() function to return without a processing error if it can't resolve a name to a pattern.
The reason for such feature is as follows:
A few programs/devices write log messages that all start the same, but then have different syntaxes, based on what they exactly log in that message. But you will be able to tell based on the common syntax at the start. An example for this would be the Cisco asa, every message starts containing something like "%ASA-4-722004:", and this number can be used to identify the syntax and data following it.
To extract the data from the variable part of said logs, you will need to make one Grok-Pattern per Log-Pattern. But said Grok-Pattern needs a rule to be applied in a pipeline. This can lead to a stage with 70 rules and more, all which look the same except a small check that a value is equal to something, and a different Grok-Pattern.
To reduce processing, and the ammount of needed rules, it would be usefull to be able to just take the value you are testing against, and instead append it to the Grok-Pattern-Name, which would result in 1 rule for every Grok-Pattern. This can be done if the existing patterns are there for every possible logging pattern, but if one is missing, it will cause a processing error, which at least adds the field with the error message to the processed log, and probably causes a performance hit from handling the error.
It's not always possible to provide a 100% coverage of the possible patterns, so a check if the corresponding Grok-Pattern exists would allow the usage of a single rule without a 100% coverage.

@kmerz kmerz self-assigned this Feb 20, 2019

@kmerz kmerz referenced this issue Feb 20, 2019
5 of 9 tasks complete

@kmerz kmerz added the in progress label Feb 20, 2019

@kmerz kmerz added this to the 3.1.0 milestone Feb 20, 2019

@bernd bernd closed this in #5699 Apr 11, 2019

bernd added a commit that referenced this issue Apr 11, 2019
Add new pipeline function "grok_exists" (#5699)
Prior to this change, a missing grok pattern would
raise a error in the pipeline processor when using the "grok"
function. But the user would like to able to make one rule
which uses a grok pattern dynamically depending on if
a grok pattern exists or not.

This change adds a new function "grok_exists" which will
return true or false depending if a grok pattern exists.
Additionally it will make a entry to the graylog-server.log
if the second argument of the function is true and
the pattern was not found.

Fixes #5689
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.