New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StreamRouter interval should be configurable #2391

Closed
pdepaepe opened this Issue Jun 21, 2016 · 1 comment

Comments

Projects
None yet
3 participants
@pdepaepe

pdepaepe commented Jun 21, 2016

Hello,

With hundreds of streams and dozens of graylog nodes, mongo load starts to be a (big) problem.
Obviously, this is mainly due to the hits done by StreamRouter every second for getting fresh rules.
As you can see on my Ganglia graph, switching this interval from 1 second to 1 minute saves me a lot of Mongo CPU.

mongo

I'm aware that this is introducing a minute of lag between stream creation and real routing and so probably need an explicit configuration directive in server.conf

@dennisoelkers dennisoelkers added this to the 2.1.0 milestone Jun 21, 2016

@dennisoelkers dennisoelkers self-assigned this Jun 21, 2016

bernd added a commit that referenced this issue Jul 14, 2016

Make several refresh intervals configurable
- Extractor filter
- Stream router
- Rules filter
- Static field filter
- In memory role permission resolver

Fixes #2391

@bernd bernd self-assigned this Jul 18, 2016

bernd added a commit that referenced this issue Jul 18, 2016

Switch stream router from polling to subscription for updates
Previously every StreamRouter instance was polling for stream changes
every second. Now it's listening on the server event bus for updates.

Stream related REST endpoints are now posting events when updating
stream configuration.

Fixes #2391

bernd added a commit that referenced this issue Jul 18, 2016

Switch extractor filter from polling to subscription for updates
Before this, every ExtractorFilter was using a cache which expired after
1 second for the extractors. So it was basically polling the extractors
every second.

Refs #2391

bernd added a commit that referenced this issue Jul 18, 2016

Switch static fields filter from polling to subscription for updates
Before this, every StaticFieldFilter instance was using a cache which
expired after 1 second. It was basically polling the static fields
configuration every second.

Refs #2391

bernd added a commit that referenced this issue Jul 18, 2016

Switch rules filter from polling to subscription for updates
Before this, every RulesFilter instance was using a cache which expired
after 1 second so it was basically polling for new rules every second.

Refs #2391
@bernd

This comment has been minimized.

Member

bernd commented Jul 19, 2016

@pdepaepe We will remove the polling for updates in the upcoming 2.1 release so we do not need options to tweak the interval. See #2496

@joschi joschi closed this in #2496 Jul 22, 2016

joschi added a commit that referenced this issue Jul 22, 2016

Switch message filters from polling to subscribing to change events (#…
…2496)

* Switch stream router from polling to subscription for updates

Previously every StreamRouter instance was polling for stream changes
every second. Now it's listening on the server event bus for updates.

Stream related REST endpoints are now posting events when updating
stream configuration.

* Switch extractor filter from polling to subscription for updates

Before this, every ExtractorFilter was using a cache which expired after
1 second for the extractors. So it was basically polling the extractors
every second.

* Switch static fields filter from polling to subscription for updates

Before this, every StaticFieldFilter instance was using a cache which
expired after 1 second. It was basically polling the static fields
configuration every second.

* Switch rules filter from polling to subscription for updates

Before this, every RulesFilter instance was using a cache which expired
after 1 second so it was basically polling for new rules every second.

Fixes #2391
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment