Using Redis

Mike Perham edited this page Nov 15, 2016 · 29 revisions

Sidekiq uses Redis to store all of its job and operational data.

By default, Sidekiq tries to connect to Redis at localhost:6379. This typically works great during development but needs tuning in production.

Using an ENV variable

You can set the Redis url using environment variables. This makes configuring Sidekiq on Heroku dead simple.

Set the REDIS_PROVIDER env var to the name of the env var containing the Redis server url. (Example with RedisGreen: set REDIS_PROVIDER=REDISGREEN_URL and Sidekiq will use the value of the REDISGREEN_URL env var when connecting to Redis.)

heroku config:set REDIS_PROVIDER=REDISTOGO_URL

You may also use the generic REDIS_URL which may be set to your own private Redis server.

Using an initializer

It is important to note that to configure the location of Redis, you must define both the Sidekiq.configure_server and Sidekiq.configure_client blocks. To do this put the following code into config/initializers/sidekiq.rb.

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://redis.example.com:7372/12' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://redis.example.com:7372/12' }
end

NOTE: Unknown parameters are passed to the underlying Redis client so any parameters supported by the driver can go in the Hash.

NOTE: The configuration hash must have symbolized keys.

Complete Control

If you need complete control when creating the Redis connection, e.g. if you are using redis-failover or Redis Sentinel, you can provide Sidekiq with a pre-built connection pool:

redis_conn = proc {
  Redis.new # do anything you want here
}
Sidekiq.configure_client do |config|
  config.redis = ConnectionPool.new(size: 5, &redis_conn)
end
Sidekiq.configure_server do |config|
  config.redis = ConnectionPool.new(size: 25, &redis_conn)
end

Note the size tuning. You'll want to ensure you have plenty of connections for the threads running in each process. Connections are created on demand so it's ok to specify a larger size (e.g. 20-30) if you aren't sure. A Sidekiq server process requires at least (concurrency + 2) connections.

Life in the Cloud

One problem with cloud-based systems like EC2 and Heroku is unpredictable network performance. You should tune your network timeouts to be a little more lenient if you are seeing occasional timeout errors, it defaults to 1 second.

config.redis = { url: 'redis://...', network_timeout: 5 }

REMEMBER: THIS IS A BANDAID You are not solving the actual cause of the slow performance.

If you are seeing Redis timeout errors, you should check your Redis latency by using the redis-cli --latency and --latency-history flags:

$ redis-cli --latency-history localhost
min: 0, max: 1, avg: 0.20 (1359 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.18 (1356 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.18 (1355 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.17 (1359 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.19 (1358 samples) -- 15.00 seconds range
min: 0, max: 1, avg: 0.18 (1353 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.19 (1357 samples) -- 15.01 seconds range

This says my average latency to localhost is 0.2ms or 200 microseconds: excellent. With users seeing odd Redis behavior, I regularly see setups with latency over 5 seconds: terrible. You can move to a different Redis provider or run your own Redis server on a dedicated machine but there's nothing Sidekiq can do if the network performance is terrible. Contact your Redis provider and ask about your available options.

Tuning

You can see Redis's config variables with the command redis-cli info.

If you have a large processing farm and are running Redis < 3.0.3, you may need to tune Redis's hz config variable. By default, hz is 10. You should increase it based on the number of connected_clients, roughly hz = connected_clients / 100. If connected_clients is 2000, set it to 20. Never set hz below 10 or above 100. If connected_clients is 2, set it to 10. If connected_clients is 20000, set it to 100. If not set properly, you can see spurious client timeouts. See #2431 for further discussion.

Memory

Redis runs best when all data fits in memory. Many people use Redis as a cache but it's important that Sidekiq be run against a Redis instance that is not configured as a cache but as a persistent store. You should use the noeviction policy so Redis doesn't drop Sidekiq's data silently. I suggest using two separate Redis instances, each configured appropriately, if you wish to use Redis for caching and Sidekiq.

Tools

  • redis-cli --help shows several useful options, including --latency-history and --bigkeys.
  • redis-rdb-tools has a few options to introspect a dumped RDB file.

Previous: Best Practices Next: Error Handling