Sharding

Sidekiq has a scalability limit: normally it can only use a single Redis server. This usually isn't a problem: Redis is really fast and on good hardware you can pump 5-10,000 jobs/sec through Redis before you'll start to hit the ceiling. For people who want to go beyond that limit you have two choices:

• Break your application into smaller applications. You can use one Redis server per application.
• Shard your application's jobs across many Redis instances.

How to Shard

The latter option is reasonably simple with the Sidekiq::Client updates in Sidekiq 3.0:

REDIS_A = ConnectionPool.new { Redis.new(...) }
REDIS_B = ConnectionPool.new { Redis.new(...) }

# Create a job in the default redis instance
SomeWorker.perform_async

# Push a job to REDIS_A using the low-level Client API
client = Sidekiq::Client.new(REDIS_A)
client.push(...)

Sidekiq::Client.via(REDIS_B) do
  # All jobs defined within this block will go to B
  SomeWorker.perform_async
end

Limitations

Sharding comes with some serious limitations:

1. The Sidekiq API does not support sharding, it only works with the default global Redis connection in Sidekiq.redis
2. The Web UI has the same restriction. You need to start a different process for each shard you want to monitor with the Web UI.
3. You need to spin up Sidekiq processes to execute jobs from each Redis instance. A Sidekiq process always only executes jobs from a single Redis instance.
4. Sharding greatly increases the complexity of your system. It's just harder to mentally track which jobs are going where and thus makes debugging harder.

I don't recommend sharding unless all other options are unavailable.