The missing unique jobs in sidekiq
Ruby HTML Shell Other
Permalink
Failed to load latest commit information.
bin Upgrade gems, ruby versions and styles Jan 2, 2017
gemfiles Deprecate sidekiq versions older than 4 Mar 27, 2017
lib Bump to 5.0.0.beta Mar 27, 2017
rails_example Upgrade gems, ruby versions and styles Jan 2, 2017
redis Don't care if the lock was released Mar 2, 2016
spec Deprecate sidekiq versions older than 4 Mar 27, 2017
.codeclimate.yml Configure codeclimate Mar 17, 2017
.csslintrc Configure codeclimate Mar 17, 2017
.dockerignore Add a dockerized app for debugging Jul 24, 2016
.editorconfig Uniqueness => Locks Sep 30, 2015
.eslintignore Configure codeclimate Mar 17, 2017
.eslintrc Configure codeclimate Mar 17, 2017
.fasterer.yml Check for performance bottle necks Nov 7, 2015
.gitignore Add real world examples Jul 24, 2016
.rspec Add mock_redis when testing in fake mode Nov 24, 2013
.rubocop.yml Upgrade gems, ruby versions and styles Jan 2, 2017
.simplecov Fix testing against older sidekiqs Oct 12, 2015
.travis.yml Deprecate sidekiq versions older than 4 Mar 27, 2017
Appraisals Deprecate sidekiq versions older than 4 Mar 27, 2017
CHANGELOG.md Deprecate sidekiq versions older than 4 Mar 27, 2017
Gemfile Upgrade gems, ruby versions and styles Jan 2, 2017
LICENSE Delicense sidekiq-unique-jobs Oct 16, 2015
README.md unique args need to be an array Aug 31, 2016
Rakefile Upgrade gems, ruby versions and styles Jan 2, 2017
circle.yml Minor circle modifications May 19, 2016
sidekiq-unique-jobs.gemspec Make gem-release a dev dependency Mar 27, 2017

README.md

SidekiqUniqueJobs Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs Build Status Code Climate Test Coverage

The missing unique jobs for sidekiq

Requirements

See https://github.com/mperham/sidekiq#requirements for what is required. Starting from 3.0.13 only sidekiq 3 is supported and support for MRI 1.9 is dropped (it might work but won't be worked on)

Version 4 Upgrade instructions

Version 4 requires redis >= 2.6.2!! Don't upgrade to version 4 unless you are on redis >= 2.6.2.

Easy path - Drop all your unique jobs before upgrading the gem!

Hard path - See below... Start with a clean slate :)

Installation

Add this line to your application's Gemfile:

gem 'sidekiq-unique-jobs'

And then execute:

$ bundle

Or install it yourself as:

$ gem install sidekiq-unique-jobs

Locking

Like @mperham mentions on this wiki page it is hard to enforce uniqueness with redis in a distributed redis setting.

To make things worse there are many ways of wanting to enforce uniqueness.

While Executing

Due to discoverability of the different lock types unique sidekiq option it was decided to use the while_executing as a default. Most people will note that scheduling any number of jobs with the same arguments is possible.

sidekiq_options unique: :while_executing

Is to make sure that a job can be scheduled any number of times but only executed a single time per argument provided to the job we call this runtime uniqueness. This is probably most useful for background jobs that are fast to execute. (See mhenrixon/sidekiq-unique-jobs#111 for a great example of when this would be right.) While the job is executing/performing no other jobs can be executed at the same time.

Until Executing

sidekiq_options unique: :until_executing

This means that a job can only be scheduled into redis once per whatever the configuration of unique arguments. Any jobs added until the first one of the same arguments has been unlocked will just be dropped. This is what was tripping many people up. They would schedule a job to run in the future and it would be impossible to schedule new jobs with those same arguments even immediately. There was some forth and back between also locking jobs on the scheduled queue and the regular queues but in the end I decided it was best to separate these two features out into different locking mechanisms. I think what most people are after is to be able to lock a job while executing or that seems to be what people are most missing at the moment.

Until Executed

sidekiq_options unique: :until_executed

This is the combination of the two above. First we lock the job until it executes, then as the job begins executes we keep the lock so that no other jobs with the same arguments can execute at the same time.

Until Timeout

sidekiq_options unique: :until_timeout

The job won't be unlocked until the timeout/expiry runs out.

Unique Until And While Executing

sidekiq_options unique: :until_and_while_executing

This lock is exactly what you would expect. It is considered unique in a way until executing begins and it is locked while executing so what differs from UntilExecuted?

The difference is that this job has two types of uniqueness:

  1. It is unique until execution
  2. It is unique while executing

That means it locks for any job with the same arguments to be persisted into redis and just like you would expect it will only ever allow one job of the same unique arguments to run at any given time but as soon as the runtime lock has been acquired the schedule/async lock is released.

Uniqueness Scope

  • Queue specific locks
  • Across all queues.
  • Timed / Scheduled jobs

Usage

All that is required is that you specifically set the sidekiq option for unique to a valid value like below:

sidekiq_options unique: :while_executing

For jobs scheduled in the future it is possible to set for how long the job should be unique. The job will be unique for the number of seconds configured (default 30 minutes) or until the job has been completed. Thus, the job will be unique for the shorter of the two. Note that Sidekiq versions before 3.0 will remove job keys after an hour, which means jobs can remain unique for at most an hour.

If you want the unique job to stick around even after it has been successfully processed then just set unique: :until_timeout.

You can also control the expiration length of the uniqueness check. If you want to enforce uniqueness over a longer period than the default of 30 minutes then you can pass the number of seconds you want to use to the sidekiq options:

sidekiq_options unique: :until_timeout, unique_expiration: 120 * 60 # 2 hours

For locking modes (:while_executing and :until_and_while_executing) you can control the expiration length of the runtime uniqueness. If you want to enforce uniqueness over a longer period than the default of 60 seconds, then you can pass the number of seconds you want to use to the sidekiq options:

sidekiq_options unique: :while_executing, run_lock_expiration: 2 * 60 # 2 minutes

Requiring the gem in your gemfile should be sufficient to enable unique jobs.

Usage with ActiveJob

Sidekiq.default_worker_options = {
  unique: :until_executing,
  unique_args: ->(args) { [ args.first.except('job_id') ] }
}

Finer Control over Uniqueness

Sometimes it is desired to have a finer control over which arguments are used in determining uniqueness of the job, and others may be transient. For this use-case, you need to define either a unique_args method, or a ruby proc.

The unique_args method need to return an array of values to use for uniqueness check.

The method or the proc can return a modified version of args without the transient arguments included, as shown below:

class UniqueJobWithFilterMethod
  include Sidekiq::Worker
  sidekiq_options unique: :until_and_while_execution,
                  unique_args: :unique_args

  def self.unique_args(name, id, options)
    [ name, options[:type] ]
  end

  ...

end

class UniqueJobWithFilterProc
  include Sidekiq::Worker
  sidekiq_options unique: :until_executed,
                  unique_args: ->(args) { [ args.first ] }

  ...

end

The previous problems with unique args being string in server and symbol in client is no longer a problem because the UniqueArgs class accounts for this and converts everything to json now. If you find an edge case please provide and example so that we can add coverage and fix it.

After Unlock Callback

If you are using :after_yield as your unlock ordering, Unique Job offers a callback to perform some work after the block is yielded.

class UniqueJobWithFilterMethod
  include Sidekiq::Worker
  sidekiq_options unique: :while_executing,

  def after_unlock
   # block has yielded and lock is released
  end
  ...
end.

Logging

To see logging in sidekiq when duplicate payload has been filtered out you can enable on a per worker basis using the sidekiq options. The default value is false

class UniqueJobWithFilterMethod
  include Sidekiq::Worker
  sidekiq_options unique: :while_executing,
                  log_duplicate_payload: true

  ...

end

Debugging

There are two ways to display and remove keys regarding uniqueness. The console way and the command line way.

Console

Start the console with the following command bundle exec jobs console.

List Unique Keys

keys '*', 100

Remove Unique Keys

del '*', 100, false the dry_run and count parameters are both required. This is to have some type of protection against clearing out all uniqueness.

Command Line

bundle exec jobs displays help on how to use the unique jobs command line.

Communication

There is a Join the chat at https://gitter.im/mhenrixon/sidekiq-unique-jobs for praise or scorn. This would be a good place to have lengthy discuss or brilliant suggestions or simply just nudge me if I forget about anything.

Testing

To enable the testing for sidekiq-unique-jobs, add require 'sidekiq_unique_jobs/testing' to your testing helper.

You can if you want use gem 'mock_redis' to prevent sidekiq unique jobs using redis.

See https://github.com/mhenrixon/sidekiq-unique-jobs/tree/master/rails_example/spec/controllers/work_controller_spec.rb for an example of how to configure sidekiq and unique jobs without redis.

If you really don't care about testing uniquness and trust we get that stuff right you can (in newer sidekiq versions) remove the client middleware.

describe "Some test" do
  before(:each) do
    Sidekiq.configure_client do |config|
      config.client_middleware do |chain|
        chain.remove SidekiqUniqueJobs::Client::Middleware
      end
    end
  end
end

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Contributors

In no particular order: