Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase the slow queue message threshold to 800ms for the MonitoringQueue #1926

Merged
merged 1 commit into from May 2, 2019

Conversation

2 participants
@shaan1337
Copy link
Member

commented May 2, 2019

Fixes #1913

GetFreshStats() spawns child processes such as 'df' or 'free'. When the child process exits, it needs to close all its file descriptors. If the ulimit for number of open files is high (RLIMIT_NOFILE), this can take some time (up to ~500ms) and will fill in the logs with slow queue messages.

Reproduction steps:
$ ulimit -n 999999
$ <start eventstore with --stats-period-sec 1>

This will fill in the logs with messages like:

[01943,20,06:53:22.764] SLOW QUEUE MSG ["MonitoringQueue"]: "GetFreshStats" - 495ms. Q: 0/1.
Increase the slow queue message threshold to 800ms for the Monitoring…
… queue:

GetFreshStats() spawns child processes such as 'df' or 'free'. When the child process exits, it needs to close all its file descriptors.
If the ulimit for number of open files is high (RLIMIT_NOFILE), this can take some time (up to ~500ms) and will fill in the logs with slow queue messages.

@shaan1337 shaan1337 requested a review from avish0694 May 2, 2019

@avish0694 avish0694 merged commit f0df2e7 into master May 2, 2019

@avish0694 avish0694 deleted the increase-monitoring-queue-slow-threshold branch May 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.