Ent Unique Jobs

If your application code creates duplicate jobs, Sidekiq's unique jobs feature makes it easy to ensure only a single copy of a job is in Redis. For instance, perhaps you create a job to sync an address change with a 3rd party system every time a form is submitted. If the form is submitted twice, you don't need to create the second job if the first job is still pending.

Enable

To activate the feature, enable it in your initializer:

# Disable uniqueness in testing, this has the potential to cause much head scratching...
Sidekiq::Enterprise.unique! unless Rails.env.testing?

Use

Declare a time period during which your job should be considered unique:

class MyWorker
  include Sidekiq::Worker
  sidekiq_options unique_for: 10.minutes

  def perform(...)
  end
end

This means that a second job can be pushed to Redis after 10 minutes or after the first job has successfully processed. If your job retries for a while, 10 minutes can pass, thus allowing another copy of the same job to be pushed to Redis. Design your jobs so that uniqueness is considered best effort, not a 100% guarantee

Jobs are considered unique based on (class, args, queue), meaning a job with the same args can be pushed to different queues.

Scheduled Jobs

The uniqueness period for a scheduled job includes the delay time. If you use MyWorker.perform_in(1.hour, ...), the uniqueness for this job will last 70 minutes (1.hour + the 10.minutes unique_for TTL). You won't be able to push the same job until it runs successfully or 70 minutes have passed.

Bypassing Uniqueness

If you really want to push a job and bypass the uniqueness check, pass false as the unique_for option with Sidekiq::Client:

Sidekiq::Client.push('class' => MyWorker, 'args' => [1,2,3], 'unique_for' => false)