StreamRouter interval should be configurable #2391

pdepaepe · 2016-06-21T12:26:37Z

Hello,

With hundreds of streams and dozens of graylog nodes, mongo load starts to be a (big) problem.
Obviously, this is mainly due to the hits done by StreamRouter every second for getting fresh rules.
As you can see on my Ganglia graph, switching this interval from 1 second to 1 minute saves me a lot of Mongo CPU.

I'm aware that this is introducing a minute of lag between stream creation and real routing and so probably need an explicit configuration directive in server.conf

- Extractor filter - Stream router - Rules filter - Static field filter - In memory role permission resolver Fixes #2391

Previously every StreamRouter instance was polling for stream changes every second. Now it's listening on the server event bus for updates. Stream related REST endpoints are now posting events when updating stream configuration. Fixes #2391

Before this, every ExtractorFilter was using a cache which expired after 1 second for the extractors. So it was basically polling the extractors every second. Refs #2391

Before this, every StaticFieldFilter instance was using a cache which expired after 1 second. It was basically polling the static fields configuration every second. Refs #2391

Before this, every RulesFilter instance was using a cache which expired after 1 second so it was basically polling for new rules every second. Refs #2391

bernd · 2016-07-19T08:35:35Z

@pdepaepe We will remove the polling for updates in the upcoming 2.1 release so we do not need options to tweak the interval. See #2496

…2496) * Switch stream router from polling to subscription for updates Previously every StreamRouter instance was polling for stream changes every second. Now it's listening on the server event bus for updates. Stream related REST endpoints are now posting events when updating stream configuration. * Switch extractor filter from polling to subscription for updates Before this, every ExtractorFilter was using a cache which expired after 1 second for the extractors. So it was basically polling the extractors every second. * Switch static fields filter from polling to subscription for updates Before this, every StaticFieldFilter instance was using a cache which expired after 1 second. It was basically polling the static fields configuration every second. * Switch rules filter from polling to subscription for updates Before this, every RulesFilter instance was using a cache which expired after 1 second so it was basically polling for new rules every second. Fixes #2391

dennisoelkers added performance feature labels Jun 21, 2016

dennisoelkers added this to the 2.1.0 milestone Jun 21, 2016

dennisoelkers self-assigned this Jun 21, 2016

bernd added a commit that referenced this issue Jul 14, 2016

Make several refresh intervals configurable

Loading status checks…

0528e98

- Extractor filter - Stream router - Rules filter - Static field filter - In memory role permission resolver Fixes #2391

bernd mentioned this issue Jul 14, 2016

Make several refresh intervals configurable #2487

Closed

bernd self-assigned this Jul 18, 2016

bernd mentioned this issue Jul 18, 2016

Switch message filters from polling to subscribing to change events #2496

Merged

joschi closed this in #2496 Jul 22, 2016

Graylog2 / graylog2-server

StreamRouter interval should be configurable #2391

StreamRouter interval should be configurable #2391

pdepaepe commented Jun 21, 2016 •

edited

bernd commented Jul 19, 2016

Graylog2 / graylog2-server

StreamRouter interval should be configurable #2391

StreamRouter interval should be configurable #2391

Comments

pdepaepe commented Jun 21, 2016 • edited

bernd commented Jul 19, 2016

pdepaepe commented Jun 21, 2016 •

edited