Advanced Options

The Sidekiq Configuration File

The Sidekiq configuration file is a YAML file that Sidekiq server uses to configure itself, by default located at config/sidekiq.yml. It is only necessary to create the file if you need to set advanced options, such as concurrency pool size, named queues, PID file location, etc. Here is an example configuration file:

---
:concurrency: 5
:pidfile: tmp/pids/sidekiq.pid
staging:
  :concurrency: 10
production:
  :concurrency: 20
:queues:
  - default
  - [myqueue, 2]

Note the use of environment-specific subsections. These values will override top-level values. If you don't use the default location, use the -C flag to tell Sidekiq where the file is:

sidekiq -C config/myapp_sidekiq.yml

Setting the Location of your Redis server

using Sidekiq's configure blocks

By default, Sidekiq assumes Redis is located at localhost:6379. This is fine for development but for many production deployments you will probably need to point Sidekiq to an external Redis server. It is important to note that to configure the location of Redis, you must define both the Sidekiq.configure_server and Sidekiq.configure_client blocks. To do this throw the following code into config/initializers/sidekiq.rb.

Sidekiq.configure_server do |config|
  config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end

Sidekiq.configure_client do |config|
  config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end

NOTE: The configuration hash must have symbolized keys.

NOTE: The :namespace parameter is optional, but recommended if Sidekiq is sharing access to a Redis database.

NOTE: Replace "7372" with the port you're using for Redis.

via ENV variable

You can also set the Redis url using environment variables.

Set the REDIS_PROVIDER env var to the key of the env var containing the redis server url. (Example with RedisGreen: set REDIS_PROVIDER=REDISGREEN_URL and Sidekiq will use the value of the REDISGREEN_URL env var when connecting to Redis.)

You may also use the generic REDIS_URL which may be set to your own private Redis server.

RedisToGo can be used with the default REDISTOGO_URL env var in Sidekiq 2.x and earlier (REDISTOGO_URL is supported for legacy reasons, the REDIS_PROVIDER and REDIS_URL vars are recommended).

Complete Control

If you need complete control when creating the Redis connection, e.g. if you are using redis-failover or Redis Sentinel, you can provide Sidekiq with a pre-built connection pool:

redis_conn = proc {
  Redis.new # do anything you want here
}
Sidekiq.configure_client do |config|
  config.redis = ConnectionPool.new(size: 5, &redis_conn)
end
Sidekiq.configure_server do |config|
  config.redis = ConnectionPool.new(size: 25, &redis_conn)
end

Queues

By default, sidekiq uses a single queue called "default" in Redis. If you want to use multiple queues, you can either specify them as arguments to the sidekiq command, or set them in the Sidekiq configuration file. Each queue can be configured with an optional weight. A queue with a weight of 2 will be checked twice as often as a queue with a weight of 1:

sidekiq -q critical,2 -q default

If you want queues always processed in a specific order, just declare them in order without weights:

sidekiq -q critical -q default -q low

You can specify a queue to use for a given worker just by declaring it:

class ImportantWorker
  include Sidekiq::Worker
  sidekiq_options :queue => :critical

  def perform(*important_args)
    puts "Doing critical work"
  end
end

You can also configure queue weights and ordering in your sidekiq config:

...
:queues:
  - [often, 7]
  - [default, 5]
  - [seldom, 3]
  - myotherqueue

Workers

Sidekiq offers a number of options for worker behavior.

• queue : use a named queue for this Worker, default 'default'
• retry : enable the RetryJobs middleware for this Worker, default true. Alternatively, you can specify the max. number of times a job is retried (ie. :retry => 3)
• backtrace : whether to save any error backtrace in the retry payload to display in web UI, can be true, false or an integer number of lines to save, default false

class HardWorker
  include Sidekiq::Worker
  sidekiq_options :queue => :crawler, :retry => false, :backtrace => true
  
  def perform(name, count)
  end
end

Default worker options can be set using Sidekiq.default_worker_options=:

Sidekiq.default_worker_options = { 'backtrace' => true }

Concurrency

You can tune the amount of concurrency in your sidekiq process. By default, one sidekiq process creates 25 Threads. If that's crushing your machine with I/O, you can adjust it down:

sidekiq -c 10

Note that ActiveRecord has a connection pool which needs to be properly configured in config/database.yml to work well with heavy concurrency. Set the pool setting to something close or equal to the number of Processors:

production:
  adapter: mysql2
  database: foo_production
  pool: 25

Heroku

If you're running on Heroku, you can't rely on the config/database.yml as that platform relies on the DATABASE_URL environment variable to determine the database connection configuration. Heroku overwrites the database.yml during slug compilation so that it reads from DATABASE_URL.

See Heroku Concurrency and DB Connections for Active Record connection pool configuration on Heroku.

Rails 3.2 and newer

As of Rails 3.2, ActiveRecord's initialization code will prefer a DATABASE_URL over the database.yml file. (See Issue #503 for a more complete explanation.) You can set a custom connection pool size for the Sidekiq server process via:

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://redis.example.com:7372/12', namespace: 'mynamespace' }

  database_url = ENV['DATABASE_URL']
  if database_url
    ENV['DATABASE_URL'] = "#{database_url}?pool=25"
    ActiveRecord::Base.establish_connection
  end
end

When we detect a DATABASE_URL environment variable we tack the new pool size onto it (this only affects the current process's environment variable). Then we force ActiveRecord::Base to re-establish a new connection with the database using the new pool size.

Rails 3.1 and older

If you're running an older (before 3.2) version of Rails you'll need to hack your DATABASE_URL environment variable so that the pool size is properly set when Heroku generates the new database.yml file:

heroku config -s | awk '/^DATABASE_URL=/{print $0 "?pool=25"}' | xargs heroku config:add

Connection Pooling

sidekiq includes the connection_pool gem which your Workers can use. With a connection pool, you can share a limited number of I/O connections among a larger number of threads.

class HardWorker
  include Sidekiq::Worker

  MEMCACHED_POOL = ConnectionPool.new(:size => 10, :timeout => 3) { Dalli::Client.new }
  def perform(args)
    MEMCACHED_POOL.with do |dalli|
      dalli.set('foo', 'bar')
    end
  end
end

This ensures that even if you have lots of concurrency, you'll only have 10 connections open to memcached per Sidekiq process.