Deployment

Deployment

Network Architecture

I recommend running 1 or more Sidekiqs per app server in your production cluster. At The Clymb, we run two Sidekiqs on each of our three app servers, for six total processes. With default concurrency of 25, this gives us 150 worker threads. All six processes talk to the same master Redis server and we just use the default queue to keep things as simple as possible.

Running a separate utility instance or using multiple, prioritized queues adds complexity where it is not needed for us.

Capistrano

Sidekiq integrates easily with Capistrano so that when you deploy your Rails app, Sidekiq will restart also. Add require 'sidekiq/capistrano' to your config/deploy.rb file and if you have custom Sidekiq configuration options put them in config/sidekiq.yml.

Note the config YAML may have an environment-specific section which overrides any global values:

---
:concurrency: 5
:pidfile: tmp/pids/sidekiq.pid
staging:
  :concurrency: 10
production:
  :concurrency: 50

By default, one Sidekiq process will be started on each app server. I prefer this because it spreads out the workload and avoids machine specialization. If you want to limit Sidekiq to a set of machines, you can define a custom Capistrano role and tell the Sidekiq recipe to use it:

set :sidekiq_role, :sidekiq
role :sidekiq, 'worker-1.acmecorp.com', 'worker-2.acmecorp.com'

Configuration options

Sidekiq's Capistrano integration can be configured with these configuration options:

set :sidekiq_cmd, "#{bundle_cmd} exec sidekiq"
set :sidekiqctl_cmd, "#{bundle_cmd} exec sidekiqctl"
set :sidekiq_timeout, 10
set :sidekiq_role, :app
set :sidekiq_pid, "#{current_path}/tmp/pids/sidekiq.pid"
set :sidekiq_processes, 1

Heroku

Using sidekiq with Heroku is simple, you need to use the (default) Cedar stack and add a Procfile to your Rails app to start a sidekiq worker process:

web: bundle exec unicorn ...
worker: bundle exec sidekiq ...

Sidekiq will automatically configure itself to use Redis-to-Go so you don't need to customize the Redis server location.

See this page for more details about Procfiles, Foreman and Heroku.

Ubuntu

I've included the Upstart config The Clymb uses to manage our Sidekiq instances on our app servers. These files go in /etc/init and will automatically startup Sidekiq when the machine is booted and can be very simply managed with [start | stop | restart] workers.