Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip reaping when queues are too large #670

Closed
francesmcmullin opened this issue Dec 9, 2021 · 2 comments · Fixed by #690
Closed

Skip reaping when queues are too large #670

francesmcmullin opened this issue Dec 9, 2021 · 2 comments · Fixed by #690

Comments

@francesmcmullin
Copy link
Contributor

Is your feature request related to a problem? Please describe.
This is a follow up to an observation in #663, when sidekiq queues are very long (e.g., more than 2,000 jobs) the reaper can take a very long time to run, more than likely hitting its timeout (depending on config).

Describe the solution you'd like
For applications with traffic/activity spikes, sidekiq queues can get very full, but they are eventually emptied. It's fine if reaping is postponed until queues are lower and there's more slack in the system.

Describe alternatives you've considered
We could play with the reaper timeout and count settings to try and find a balance such that reaping can continue even under heavy load, but for our use case it's not a problem if reaping is delayed during heavy activity.

Additional context
All my observations relate to the ruby reaper, it is possible that the lua reaper has no trouble with big queues

@mhenrixon
Copy link
Owner

Thank you for the issue, I'll do some thinking on this but I will likely not be able to achieve much until after Christmas.

I am moving my family across Germany on Monday.

@francesmcmullin
Copy link
Contributor Author

Of course! You've helped a huge amount already. I hope the move goes really well. Thank you =)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants