An ergonomic wrapper API for Sidekiq Pro Batches
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib/sidekiq
spec
.gitignore
.rspec
.rubocop.yml
CHANGELOG.md
Gemfile
LICENSE.txt
README.md
Rakefile
sidekiq-clutch.gemspec

README.md

Sidekiq::Clutch

Sidekiq::Clutch is the API I always wanted when working with Sidekiq Pro Batches. So I built it!

Features:

  • Add jobs to run in series or parallel or mix-and-match however you wish.
  • Pass results from one job onto the next job in series.
  • If running jobs in parallel, pass all the results to the following job in series.

Installation

Add this line to your application's Gemfile:

gem 'sidekiq-clutch'

And then execute:

$ bundle

Or install it yourself as:

$ gem install sidekiq-clutch

Usage

Include the Sidekiq::Clutch::Worker mixin in your job class and write your #perform method as usual. For example:

class MyJob1
  include Sidekiq::Worker
  include Sidekiq::Clutch::Worker

  def perform
    # do stuff
  end
end

Now, enqueue your jobs using a new Sidekiq::Clutch instance, like this:

clutch = Sidekiq::Clutch.new
clutch.jobs << MyJob1
clutch.jobs << [MyJob2, 'arg1', 'arg2']
clutch.parallel do
  clutch.jobs << [MyJob3, 3]
  clutch.jobs << [MyJob3, 4]
  clutch.jobs << [MyJob3, 5]
end
clutch.jobs << MyFinalizerJob
clutch.engage

The jobs will run in this order:

  • First, MyJob1 will run.
  • Second, MyJob2 with args 'arg1' and 'arg2' will run.
  • Third, in parallel, three jobs of MyJob3, each with args 3, 4, and 5 will run.
  • Last, the MyFinalizerJob will run.

Results from Prior Jobs

You can access the results from the previous job in the series using the previous_results method:

class MyJob1
  include Sidekiq::Worker
  include Sidekiq::Clutch::Worker

  def perform
    'This string will be passed to the next job in the series'
  end
end

class MyJob2
  include Sidekiq::Worker
  include Sidekiq::Clutch::Worker

  def perform(arg1, arg2)
    puts "results of the previous job (MyJob1):"
    p previous_results # => ["This string will be passed to the next job in the series"]
  end
end

If the previous step in a series was a batch of parallel jobs, then previous_results will be an array containing all the results from all the parallel jobs.

Some notes:

  • Results are always in an array, even if there was only a single prior job run in series.
  • Results are serialized as JSON and temporarily stored in Redis, so only return small-ish values.

Handling Failure

When jobs fail, Sidekiq will retry them as usual. See here for details on how Sidekiq handles job failure and retries.

Jobs next in a series will not be enqueued by Clutch as long as a prior job is failing.

If you wish to do something when a job in your batch fails, set on_failure like this:

clutch = Sidekiq::Clutch.new
clutch.on_failure = MyFailureHandlerJob
# add jobs here...
clutch.engage

MyFailureHandlerJob.new.perform(status) will be called if one of the following scenarios occur:

  1. A job in series fails.
  2. One or more jobs in parallel fail and the rest complete.

Note: retries will continue to occur even after your failure handler job is called.

Nested Batches

You can nest Clutch instances too!

class MyNestedJob
  include Sidekiq::Worker
  include Sidekiq::Clutch::Worker

  def perform
    clutch = Sidekiq::Clutch.new(batch)
    clutch.jobs << [MyJob2, 'arg1', 'arg2']
    clutch.jobs << [MyJob3, 3]
    clutch.engage
    'result from MyNestedJob'
  end
end

clutch = Sidekiq::Clutch.new
clutch.jobs << MyJob1
clutch.jobs << MyNestedJob
clutch.jobs << MyFinalizerJob
clutch.engage

Setting the Queue

You can set the queue for jobs (overriding any queue specified in the job class itself) and callbacks by setting it on Clutch, like this:

clutch = Sidekiq::Clutch.new
clutch.queue = 'critical'
# add jobs here...
clutch.engage

Note about the sidekiq-batch gem

I tested this with the third-party gem sidekiq-batch, which purports to be a drop-in replacement for Sidekiq Pro batches. However, that gem seems to not support nested callbacks, which my gem relies on. I'm sorry, this gem will not work with the sidekiq-batch gem -- it only works with sidekiq-pro.

License

The gem is available as open source under the terms of the MIT License.