-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filebeat suggestion - allowing for something like multiline.flush_pattern? #3964
Comments
We discussed in the past about a "start/end" pattern but went in the first implementation only for one pattern as it was much simpler to do. I definitively think there is value in having something similar as you described above. Here you can also find the initial discussion about multiline and what options we discussed: https://github.com/elastic/filebeat/issues/301 It reminds me also of #3852 Not sure if the two feature requests could be combined into one, means it is not necessarly only mutiline specific. |
The first link you posted is from a private issue, so i can't access it. Regarding the #3852 issue about one-line logs, I'm not sure that this flush-pattern option would solve this - dosen't the initial line-matching logic happen outside the multiline module? I've managed to add the flush-pattern code pretty simply into the current multiline code, and will create a PR soon - Just need to test it a bit more, only TDDd it so far. |
This allows for specifying a regex, which will flush the current multiline, thus ending the current multiline. Useful for using multiline to capture application events with 'start' and 'end' lines. Example configuration multiline.pattern: 'start' multiline.negate: true multiline.match: after multiline.flush_pattern: 'end' (elastic#3964)
This allows for specifying a regex, which will flush the current multiline, thus ending the current multiline. Useful for using multiline to capture application events with 'start' and 'end' lines. Example configuration multiline.pattern: 'start' multiline.negate: true multiline.match: after multiline.flush_pattern: 'end' (#3964)
I'm using multiline for merging my log-lines together to form application-centric events, which can span multiple events.
I feel like it could be a useful addition, to be able to specify a specific pattern, for when a multiline should stop and flush the event. Right now it seems to be only when either; a new multiline starts, the max-lines count is reached or the timeout is reached.
This would make it possible to flush an event right when it ends - whereas right now the last event i receive, will first be flushed when either one the timeout/max-length is reached, or when a new one begins.
What do you think?
Now i having looked at the codebase (not that familiar with go, but i might give it a try if you suggest me to), but I'm thinking that, being that theres already the other kind of flush-guards in place, it would seem conceptually like a small task to add this?
The text was updated successfully, but these errors were encountered: