Deployment

Network Architecture

I recommend running 1 or more Sidekiqs per app server in your production cluster. At The Clymb, we run two Sidekiqs on each of our three app servers, for six total processes. With default concurrency of 25, this gives us 150 worker threads. All six processes talk to the same master Redis server and we just use the default queue to keep things as simple as possible.

Running a separate utility instance or using multiple, prioritized queues adds complexity where it is not needed for us.

Capistrano

Sidekiq integrates easily with Capistrano so that when you deploy your Rails app, Sidekiq will restart also. For Capistrano v2, add require 'sidekiq/capistrano' to your config/deploy.rb file. For Capistrano v3 add require 'sidekiq/capistrano' to your Capfile file.

Example Capistrano v3 Capfile

# Load DSL and Setup Up Stages
require "capistrano/setup"

# Includes default deployment tasks
require "capistrano/deploy"

# Includes tasks from other gems included in your Gemfile
#
# require "capistrano/rvm"
require "sidekiq/capistrano"

# Loads custom tasks from "lib/capistrano/tasks" if you have any defined.
Dir.glob("lib/capistrano/tasks/*.cap").each { |r| import r }

Sidekiq.yml

If you have custom Sidekiq configuration options put them in config/sidekiq.yml. Note the config YAML may have an environment-specific section which overrides any global values:

---
:concurrency: 5
:pidfile: tmp/pids/sidekiq.pid
staging:
  :concurrency: 10
production:
  :concurrency: 50

By default, one Sidekiq process will be started on each app server. I prefer this because it spreads out the workload and avoids machine specialization. If you want to limit Sidekiq to a set of machines, you can define a custom Capistrano role and tell the Sidekiq recipe to use it:

set :sidekiq_role, :sidekiq
role :sidekiq, 'worker-1.acmecorp.com', 'worker-2.acmecorp.com'

Configuration options

Sidekiq's Capistrano integration can be configured with these configuration options:

set(:sidekiq_cmd) { "#{bundle_cmd} exec sidekiq" }
set(:sidekiqctl_cmd) { "#{bundle_cmd} exec sidekiqctl" }
set(:sidekiq_timeout) { 10 }
set(:sidekiq_role) { :app }
set(:sidekiq_pid) { "#{current_path}/tmp/pids/sidekiq.pid" }
set(:sidekiq_processes) { 1 }

Heroku

Using sidekiq with Heroku is simple, you need to use the (default) Cedar stack and add a Procfile to your Rails app to start a sidekiq worker process:

web: bundle exec unicorn ...
worker: bundle exec sidekiq ...

Sidekiq will automatically configure itself to use Redis-to-Go so you don't need to customize the Redis server location.

See this page for more details about Procfiles, Foreman and Heroku.

Daemonization

I strongly recommend people use a process supervisor like runit or upstart to manage Sidekiq (or any other server daemon) versus something simpler like nohup or the -d flag. This ensures Sidekiq will immediately restart if it crashes for some reason.

I've included the Upstart config The Clymb uses to manage our Sidekiq instances on our app servers. These files go in /etc/init and will automatically startup Sidekiq when the machine is booted. The Sidekiq process group can be very simply managed with [start | stop | restart] workers.