Ent Rolling Restarts

Do you have long-running jobs which might take minutes or hours to finish? Do you like to deploy frequently? Sidekiq's traditional TSTP+TERM shutdown process can cause jobs to rerun, especially if they are long-running.

Sidekiq Enterprise 1.8.0 adds support for rolling restarts. In this mode, a Sidekiq process will quiet immediately but will not exit until all jobs are complete. There is no limit to the time it can continue running. Upon signalling a rolling restart, a new process will be started to pick up new jobs.

How

einhorn is a special process manager which manages the rolling restart. Use it to run sidekiq or sidekiqswarm:

einhorn -m10 -- bundle exec sidekiqswarm -q critical -q default -q low

Send the USR2 signal to the einhorn process after deploying your new codebase to trigger a rolling restart. If the deploy fails, don't send the signal; the old processes will keep running as normal.

If the new Sidekiq process fails within 10 seconds (i.e. if the new codebase is buggy), einhorn will not stop the old process.

Integration

Modern init systems implement restart as explicit start and stop operations. You'll want to keep start and stop available for situations where a rolling restart is not possible, like major database migrations.

I recommend you update your systemd/upstart integration so that reload triggers a rolling restart.

Ideally you'll have two types of deploys: the default, "minor" deploy which reloads Sidekiq and a "major" deploy which stops Sidekiq completely, runs any migrations and then starts it back up.

# For Upstart users
# /etc/init/sidekiq.conf
reload signal USR2

# For systemd users
# /etc/systemd/system/sidekiq.service
ExecReload=kill -USR2 $MAINPID

Then a rolling restart is as simple as:

# systemd
servicectl reload sidekiq
# upstart
initctl reload sidekiq

Notes

• Old and new processes will be running at the same time. You will need to consider compatibility and take care around database migrations.
• Rolling restarts work with both the sidekiq and sidekiqswarm binaries.