-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance issue after upgrading from 2.3.x to 5.6.3 (70k -> 8k eps) #8560
Comments
Thanks for reporting this. After investigating this, the cause is the new timeout enforcement feature in grok. While this is quite useful, it is inherently expensive. I think we should add a flag to disable that if users want it. |
I verified this, BTW, by commenting the watchdog code in/out in the A few attempted optimizations only yielded a 5k/s improvement. This needs to be researched and improved in a more methodical way than my own ad-hoc experiment. |
@andrewvc Thanks for looking at this. What watchdog code did you comment out - was this in grok plugin or in core? Would setting timeout_millis => 0 have the same affect? One thing that was really surprising in perf testing was that single grok with 20 patterns is much worse than 20 groks with one pattern. Is this a seperate issue or still related to the timeout enforcement? |
@andrewvc see logstash-plugins/logstash-filter-grok#125, could make it more than twice as fast for the posted config without changes to functionality. If we want more, I'd go with your suggestion to simply bypass the timeout logic when the timeout is set to |
@andrewvc thanks for your feedback. @original-brownbear thanks for digging into this :). I rerun the benchmark with your patch, and the results look great:
|
@marioluan nice, thanks a lot for benchmarking this on I admit I only ran it against |
@original-brownbear rerun the benchmark with your new patch (logstash-plugins/logstash-filter-grok@1601d93).
|
@marioluan ah nice, thanks :) |
hey guys, do you have any expectation when this will be fixed/pushed/merged? |
@marioluan we have just published version 3.4.5 of the grok plugin, that should correctly handle this issue. |
That's wonderful news! |
How to install this new version ? bin/logstash-plugin install --version 3.4.5 logstash-filter-grokValidating logstash-filter-grok-3.4.5 |
@jmreymond the version is |
After upgrading from 2.3.4 to 5.6.3, I noticed two performance issues:
Version:
5.6.3
Operating System:
RHEL 5.3
Config File :
Run logstash with the provided configuration for 5 minutes and check
rate_1m
.The text was updated successfully, but these errors were encountered: