Cleaning up after hard shutdowns #2071
Comments
|
Heroku is special: I can't tell the PID of the process so I have no way to accurately heartbeating. Normally the heartbeat is associated with the PID: process crashes, another process starts with another PID, old heartbeat ceases and job data expires in Redis. With heroku, dyno 1 crashes, it's restarted so the heartbeat immediately restarts for dyno 1, data never expires. One possible fix: have each process reset its job state when it starts up so it can't inherit old job data. |
|
Could we also just associate the heartbeat with a UUID that's generated when the process first starts up? |
|
Solid idea. Let me think about that. |
|
Would you be open to me putting together a preliminary PR for this? Given that the bug interferes with our autoscaling and can impact our Heroku costs significantly, we're definitely open to spending some time on it ourselves. |
|
Go for it.
|
In environments with deterministic hostnames and PIDs (i.e. Heroku), the current system for constructing a process's identity string produces collisions (e.g. a new process can assume the identity of one that has died). In order to address this, we add a UUID to the identity string. Issue sidekiq#2071.
In environments with deterministic hostnames and PIDs (i.e. Heroku), the current system for constructing a process's identity string produces collisions (e.g. a new process can assume the identity of one that has died). In order to address this, we add a random number to the identity string. Issue sidekiq#2071.
In environments with deterministic hostnames and PIDs (i.e. Heroku), the current system for constructing a process's identity string produces collisions (e.g. a new process can assume the identity of one that has died). In order to address this, we add a random string to the identity string. Issue sidekiq#2071.
In environments with deterministic hostnames and PIDs (i.e. Heroku), the current system for constructing a process's identity string produces collisions (e.g. a new process can assume the identity of one that has died). In order to address this, we add a random string to the identity string. Issue sidekiq#2071.
|
Addressed in #2113 |
Hi there,
We're running Sidekiq v3.3.0 with sidekiq-pro v1.9.1 on Heroku and are having some issues with workers that don't get a chance to cleanup after themselves. Our workers get hard-shutdown if either 1) the dyno shuts down abruptly without sending the SIGTERM or 2) the worker fails to exit within the 10s period. If jobs are actively running during one of these hard shutdowns, they remain permanently on the busy page until we manually clear the data structures in Redis. Aside from being annoying in the interface, these phantom jobs are also problematic because they interfere with our processes for auto-scaling dynos.
Two questions:
We're working to make these hard shutdowns less common (by talking with heroku and by getting our app to shutdown faster). Since I doubt that we'll ever eliminate them completely, however, we'd definitely appreciate thoughts on how to tackle them from the Sidekiq perspective as well.
Thanks so much!
The text was updated successfully, but these errors were encountered: