Ent Rolling Restarts

Mike Perham edited this page Nov 19, 2018 · 10 revisions

Do you have long-running jobs which might take minutes or hours to finish? Do you like to deploy frequently? Sidekiq's traditional TSTP+TERM shutdown process can cause jobs to rerun, especially if they are long-running.

Sidekiq Enterprise 1.7.0 adds support for rolling restarts. In this mode, a Sidekiq process will quiet immediately but will not exit until all jobs are complete. There is no limit to the time it can continue running. Upon signalling a rolling restart, a new process will be started to pick up new jobs.

How

einhorn is a special process manager which manages the rolling restart. Use it to run sidekiq or sidekiqswarm:

einhorn -m10 -- bundle exec sidekiqswarm [...cli args...]

To trigger a rolling restart, you tell einhorn to upgrade the binary after deploying the new code:

einhornsh --execute upgrade

If the new process is still running after 10 seconds, the old process will get USR2 and gracefully shutdown.

Integration

Modern init systems implement restart as explicit start and stop operations. You can use reload to trigger a rolling restart.

Ideally you'll have two types of deploys: the default, "minor" deploy which reloads Sidekiq and a "major" deploy which stops Sidekiq completely, runs any migrations and then starts it back up.

# For Ubuntu systemd users
# /lib/systemd/system/sidekiq.service
ExecReload=/usr/local/bin/bundle exec einhornsh --execute upgrade

Then a rolling restart is as simple as:

# systemd
systemctl reload sidekiq

Notes

  • Old and new processes will be running at the same time so memory usage will increase.
  • Consider compatibility between old and new code versions; take care around database migrations.
  • Rolling restarts work with both the sidekiq and sidekiqswarm binaries.
  • Einhorn has lots of options which might be useful for various edge cases, e.g. if your app uses Capistrano to deploy.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.