Mike Perham edited this page Apr 23, 2016 · 44 revisions

Network Architecture

I recommend running 1 or more Sidekiqs per app server in your production cluster. At The Clymb, we run two Sidekiqs on each of our three app servers, for six total processes. With default concurrency of 25, this gives us 150 worker threads. All six processes talk to the same master Redis server and we use high, default and low queues to keep things as simple as possible.

Running separate machines for Sidekiq or using many different queues adds complexity where it is not needed for us.



To safely shut down Sidekiq, you need to send it the USR1 signal as early as possible in your deploy process and the TERM signal as late as possible. USR1 tells Sidekiq to stop pulling new work and finish all current work. TERM tells Sidekiq to exit within N seconds, where N is set by the -t timeout option and defaults to 8. Using USR1+TERM in your deploy process gives your jobs the maximum amount of time to finish before exiting.

If any jobs are still running when the timeout is up, Sidekiq will push those jobs back to Redis so they can be rerun later.


Use the capistrano-sidekiq gem (github). Integrated support has been removed.


Using sidekiq with Heroku is simple, add a Procfile to your Rails app to start a sidekiq worker process:

web: bundle exec puma ...
worker: bundle exec sidekiq ...

To connect to your Redis addon, you'll need to tell Sidekiq which environment variable name to use. Set heroku config:set REDIS_PROVIDER=REDISTOGO_URL env variable for it to work.

Keep in mind that Heroku puts a hard limit of 10 seconds on a process restart and you cannot send a USR1 signal so restarting Sidekiq gracefully when running long running jobs is tough. Sidekiq will push the unfinished jobs back to Redis; make sure your jobs are idempotent so it can restart them when the process starts back up.

See this page for more details about Procfiles, Foreman and Heroku.


I strongly recommend people not to use the -d flag but instead use a process supervisor like systemd or upstart to manage Sidekiq (or any other server daemon). This ensures Sidekiq will immediately restart if it crashes for some reason.

Here's an example Upstart config used to manage Sidekiq instances. These files go in /etc/init and will automatically startup Sidekiq when the machine is booted. The Sidekiq process group can be very simply managed with [start | stop | restart] workers.


Sidekiq fires process lifecycle events when starting up and shutting down:

Sidekiq.configure_server do |config|
  # runs after your app has finished initializing but before any jobs are dispatched.
  config.on(:startup) do
  config.on(:quiet) do
    puts "Got USR1, stopping further job processing..."
  config.on(:shutdown) do
    puts "Got TERM, shutting down process..."

This can be extremely useful if you want to start/stop your own threads/actors.

Previous: Delayed Extensions Next: Monitoring