Pro Reliability Server

Sidekiq does what it can to not lose jobs. When it shuts down, it will push back any unfinished jobs to Redis. 99% of the time, that's sufficient. But it is limited: jobs are stored in-process while executing so if the process crashes or network connectivity goes down, the job can be lost.

To prevent job loss by Sidekiq, the job must remain in Redis while Sidekiq executes it. Sidekiq Pro provides two different algorithms to do just that. To activate one, add this to your initializer:

ONE_HOUR = 3600 # this is the default
Sidekiq.configure_server do |config|
  config.reliable_fetch!
  config.timed_fetch! ONE_HOUR
end

reliable_fetch

This is the algorithm that Sidekiq Pro has provided from Day 1. It uses the rpoplpush command and stores jobs within a private queue for each process while executing.

Pros

• It scales to 10,000+ jobs/sec because it uses O(1) operations.
• It's old and battle tested.

Cons

• Requires stable hostnames and a unique index per-process
• Does not work well with Heroku, Docker, Amazon's ECS or Elastic Beanstalk

Good choice if you are running in the traditional manner on your own servers, virtual or physical. Avoid if you are using containers or a PaaS like Heroku.

timed_fetch

This is a new algorithm introduced in Sidekiq Pro v3.1. It uses Lua and stores jobs within a "pending" area with a timeout. If the job execution is not finished and acknowledged by the client within that timeout period, the job can be pushed back onto the queue for another process to pick up.

Pros

• No special configuration or specialization required
• Works in every deployment environment, containers or not

Cons

• Less scalable because it uses O(log N) operations.
• New, unproven

Good choice for anyone processing less than 10M jobs/day or wanting to use containers.