Signals

Juanito Fatas edited this page Apr 22, 2015 · 22 revisions

Sidekiq responds to several signals.

TTIN

Sidekiq will respond to SIGTTIN by printing backtraces for all threads in the process to the logfile. This is useful for debugging if you have a Sidekiq process that appears to be dead or stuck.

kill -TTIN [worker_pid]

USR1

USR1 tells Sidekiq it will be shutting down at some point in the near future. It will stop accepting new work but continue working on current messages. Use USR1+TERM to guarantee shutdown within a time period. Best practice is to send USR1 at the start of a deploy and TERM at the end of a deploy.

USR2

Sidekiq will reopen all logfiles that have been rotated using logrotate (without copytruncate) or similar tools. This behavior is the same behavior as Unicorn's USR1 response.

TERM

TERM signals that Sidekiq should shut down within the -t timeout option given at start-up. It will stop accepting new work, but continue working on current messages (as with USR1). Any workers that do not finish within the timeout are forcefully terminated and their messages are pushed back to Redis to be executed again when Sidekiq starts up. The timeout defaults to 8 seconds since all Heroku processes must exit within 10 seconds.

Capistrano

The capistrano integration sends USR1 at the very start of the deploy and sends TERM at the end of the deploy in order to give the maximum amount of time for Sidekiq workers to finish their work. Since a Capistrano deploy might take 60 seconds or more, often the timeout defaults are fine because all the workers have finished their current job by the time TERM is signalled.

The Sidekiq TERM timeout is set in config/sidekiq.yml or with the -t parameter. Sidekiq will do its best to exit by this timeout when it receives TERM.

:timeout: 30

sidekiqctl

Sidekiq ships with a control executable for shutting it down.

sidekiqctl [quiet|stop] [path-to-pidfile] [deadline_timeout]

The Capistrano integration does this:

# start of deploy
# quiet sends USR1
sidekiqctl quiet [pidfile]
# ... deploy happens ...
# stop sends TERM with a hard deadline to kill -9
sidekiqctl stop [pidfile] [deadline_timeout]

Alternatively you can do the same thing with a single stop command but you'll want to give your worker time to finish their current jobs.

sidekiqctl stop [pidfile] 60

This sends TERM, waits up to 60 seconds and then will kill -9 the Sidekiq process if it has not exited by then. Keep in mind the deadline timeout is the amount of time sidekiqctl will wait before running kill -9 on the Sidekiq process. It's different from Sidekiq's timeout and should be set longer so Sidekiq is not killed before it has a chance to exit normally.

Previous: Logging Next: FAQ