Pro Reliability Client

Mike Perham edited this page Mar 16, 2016 · 2 revisions

When the Sidekiq client pushes a job to Redis, it just assumes the network call will work. There's no error handling so any exception will trickle up into your app and cause a 500 error. The Sidekiq Pro client offers additional reliability by locally enqueueing the job for delivery once the network connection is successfully re-established.

There are a few limitations:

  • the local queue is per-process and in-memory so if the client process is restarted, the jobs are lost.
  • only the last 10,000 pushes are saved, to prevent a long-lasting outage from filling all memory
  • the local queue doesn't work with Batches so any Redis network issues when creating a batch will still cause an exception and fail.
  • the local queue is drained the next time a job is pushed. If a push fails and then the process is idle for hours, that job will be unexpectedly delayed. Ideally your production system has enough traffic to ensure timely drainage.

You can activate "reliable push" in your sidekiq initializer:

# This should not go in a Sidekiq.configure_{client,server} block.
Sidekiq::Client.reliable_push! unless Rails.env.test?

You don't want reliable push during testing because you don't want it to swallow unexpected errors and cause your test suite to pass despite problems.