Signals

Mike Perham edited this page Feb 22, 2018 · 27 revisions

Sidekiq responds to several signals.

TTIN

Sidekiq will respond to SIGTTIN by printing backtraces for all threads in the process to the logfile. This is useful for debugging if you have a Sidekiq process that appears to be dead or stuck.

kill -TTIN [worker_pid]

TSTP

TSTP tells Sidekiq to "quiet" as it will be shutting down at some point in the near future. It will stop fetching new jobs but continue working on current jobs. Use TSTP + TERM to guarantee shutdown within a time period. Best practice is to send TSTP at the start of a deploy and TERM at the end of a deploy.

Note you still need to send TERM to actually exit the Sidekiq process.

The quiet signal used to be USR1 but was changed to TSTP in Sidekiq 5.0.

USR2

Sidekiq will reopen all logfiles that have been rotated using logrotate (without copytruncate) or similar tools. This behavior is the same behavior as Unicorn's USR1 response. I do not recommend using this, your daemon should be logging to stdout and letting upstart/systemd handle your logs for you.

Sidekiq Enterprise uses USR2 as the trigger for a rolling restart.

TERM

TERM signals that Sidekiq should shut down within the -t timeout option given at start-up. It will stop accepting new work, but continue working on current messages (as with TSTP). Any workers that do not finish within the timeout are forcefully terminated and their messages are pushed back to Redis to be executed again when Sidekiq starts up. The timeout defaults to 8 seconds since all Heroku processes must exit within 10 seconds.

As of 2017, Heroku processes can now take up to 30 seconds to shutdown. It is recommended you set Sidekiq's shutdown timeout to 25 seconds to give your jobs the maximum amount of time to shutdown: -t 25

Capistrano

The capistrano integration sends TSTP at the very start of the deploy and sends TERM at the end of the deploy in order to give the maximum amount of time for Sidekiq workers to finish their work. Since a Capistrano deploy might take 60 seconds or more, often the timeout defaults are fine because all the workers have finished their current job by the time TERM is signalled.

The Sidekiq TERM timeout is set in config/sidekiq.yml or with the -t parameter. Sidekiq will do its best to exit by this timeout when it receives TERM.

:timeout: 30

sidekiqctl

Sidekiq ships with a control executable for shutting it down.

sidekiqctl [quiet|stop] [path-to-pidfile] [deadline_timeout]

The Capistrano integration does this:

# start of deploy
# quiet sends TSTP
sidekiqctl quiet [pidfile]
# ... deploy happens ...
# stop sends TERM with a hard deadline to kill -9
sidekiqctl stop [pidfile] [deadline_timeout]

Alternatively you can do the same thing with a single stop command but you'll want to give your worker time to finish their current jobs.

sidekiqctl stop [pidfile] 60

This sends TERM, waits up to 60 seconds and then will kill -9 the Sidekiq process if it has not exited by then. Keep in mind the deadline timeout is the amount of time sidekiqctl will wait before running kill -9 on the Sidekiq process. It's different from Sidekiq's timeout and should be set longer so Sidekiq is not killed before it has a chance to exit normally.

Previous: Logging Next: FAQ

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.