/graylog2-server

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StreamRouter interval should be configurable #2391

Closed
pdepaepe opened this Issue Jun 21, 2016 · 1 comment

Comments

Assignees

@bernd bernd

@dennisoelkers dennisoelkers

Labels
feature performance
Projects
None yet
Milestone
  2.1.0
3 participants
@pdepaepe @bernd @dennisoelkers
@pdepaepe

pdepaepe commented Jun 21, 2016 •
edited

Hello,

With hundreds of streams and dozens of graylog nodes, mongo load starts to be a (big) problem.
Obviously, this is mainly due to the hits done by StreamRouter every second for getting fresh rules.
As you can see on my Ganglia graph, switching this interval from 1 second to 1 minute saves me a lot of Mongo CPU.

mongo

I'm aware that this is introducing a minute of lag between stream creation and real routing and so probably need an explicit configuration directive in server.conf

@dennisoelkers dennisoelkers added performance feature labels Jun 21, 2016

@dennisoelkers dennisoelkers added this to the 2.1.0 milestone Jun 21, 2016

@dennisoelkers dennisoelkers self-assigned this Jun 21, 2016

bernd added a commit that referenced this issue Jul 14, 2016

@bernd
Make several refresh intervals configurable 
- Extractor filter
- Stream router
- Rules filter
- Static field filter
- In memory role permission resolver

Fixes #2391
Loading status checks…
0528e98

@bernd bernd referenced this issue Jul 14, 2016

Closed

Make several refresh intervals configurable #2487

@bernd bernd self-assigned this Jul 18, 2016

bernd added a commit that referenced this issue Jul 18, 2016

@bernd
Switch stream router from polling to subscription for updates 
Previously every StreamRouter instance was polling for stream changes
every second. Now it's listening on the server event bus for updates.

Stream related REST endpoints are now posting events when updating
stream configuration.

Fixes #2391
b63403c

bernd added a commit that referenced this issue Jul 18, 2016

@bernd
Switch extractor filter from polling to subscription for updates 
Before this, every ExtractorFilter was using a cache which expired after
1 second for the extractors. So it was basically polling the extractors
every second.

Refs #2391
3213d70

bernd added a commit that referenced this issue Jul 18, 2016

@bernd
Switch static fields filter from polling to subscription for updates 
Before this, every StaticFieldFilter instance was using a cache which
expired after 1 second. It was basically polling the static fields
configuration every second.

Refs #2391
c84a99a

bernd added a commit that referenced this issue Jul 18, 2016

@bernd
Switch rules filter from polling to subscription for updates 
Before this, every RulesFilter instance was using a cache which expired
after 1 second so it was basically polling for new rules every second.

Refs #2391
Loading status checks…
ac5a23f

@bernd bernd referenced this issue Jul 18, 2016

Merged

Switch message filters from polling to subscribing to change events #2496

@bernd

This comment has been minimized.

Sign in to view
Member

bernd commented Jul 19, 2016

@pdepaepe We will remove the polling for updates in the upcoming 2.1 release so we do not need options to tweak the interval. See #2496

@joschi joschi closed this in #2496 Jul 22, 2016

joschi added a commit that referenced this issue Jul 22, 2016

@bernd @joschi
Switch message filters from polling to subscribing to change events (#… 
…2496)

* Switch stream router from polling to subscription for updates

Previously every StreamRouter instance was polling for stream changes
every second. Now it's listening on the server event bus for updates.

Stream related REST endpoints are now posting events when updating
stream configuration.

* Switch extractor filter from polling to subscription for updates

Before this, every ExtractorFilter was using a cache which expired after
1 second for the extractors. So it was basically polling the extractors
every second.

* Switch static fields filter from polling to subscription for updates

Before this, every StaticFieldFilter instance was using a cache which
expired after 1 second. It was basically polling the static fields
configuration every second.

* Switch rules filter from polling to subscription for updates

Before this, every RulesFilter instance was using a cache which expired
after 1 second so it was basically polling for new rules every second.

Fixes #2391
Loading status checks…
e6b859c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment