-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Need details of what happens to running jobs during Shutdown #6213
Comments
I don't document this because it starts to get into implementation detail. Anything I document becomes behavior I have to support and maintain backwards compatibility with. I always advise people to read the code itself, the How Sidekiq Works blog post talks a bit about job fetch and shutdown too.
|
Thanks for the info on this. I think #2 might be worth the documentation because the behavior is effectively "jobs disappear if they did not stop in time" because that Sidekiq::Shutdown exception won't be one that any standard handlers deal with. |
Could you explain more? Long running jobs are pushed back to Redis by |
@mperham - sorry, I thought I responded. There are two reasons I say that.
Closely related in all this is that we just get a ton of questionable behaviors around Sidekiq in shutdown situations. For example, we had an instance get hard-killed earlier today, and 2 hours later the jobs it was doing are still showing in the UI as being in progress. The jobs were indeed re-enqueued and are happening on another machine as well, but even 2 hours later the working set status is incorrect. |
The jobs showing in Busy for hours later is because of the hard kill. Sidekiq's heartbeat is not fully transactional in order to minimize its runtime overhead, it uses sidekiq/lib/sidekiq/launcher.rb Lines 151 to 157 in 0507f0b
Here's a simple fix (-1 means "no expiry" in Redis terms):
|
...and that leakage is now fixed on main. |
This is selfishly for Sidekiq Pro but I think it applies to all the others.
On https://github.com/sidekiq/sidekiq/wiki/Deployment, it says
But what does that actually mechanically mean?
The text was updated successfully, but these errors were encountered: