Really Complex Workflows with Batches

Sidekiq Pro's Batches feature can handle job workflows of any complexity. This page shows how to implement a complex workflow given to me by one Sidekiq Pro customer.

The workflow looks like this, where jobs are blue circles and purple boxes hold jobs which can execute in parallel. All jobs within a purple box must succeed before the workflow can move "down".

Let's call this workflow the Order workflow. Perhaps it represents the series of steps necessary to ship a customer's order.

First of all, we're going to create an overall Batch to represent the entire workflow and then create a child batch to represent the first step in the workflow. That child batch will use a success callback to schedule step 2 in the workflow:

order = ...
overall = Sidekiq::Batch.new
overall.on(:success, 'OrderCallbacks#shipped', 'oid' => order.id)
overall.description = "Order #{order.id}"
overall.jobs do
  step1 = Sidekiq::Batch.new
  step1.on(:success, 'OrderCallbacks#step1_done', 'oid' => order.id)
  step1.jobs do
    A.perform_async(order.id)
  end
end

class OrderCallbacks
  def step1_done(status, options)
    oid = options['oid']
    overall = Sidekiq::Batch.new(status.parent_bid)
    overall.jobs do
      step2 = Sidekiq::Batch.new
      step2.on(:success, 'OrderCallbacks#step2_done', 'oid' => order.id)
      step2.jobs do
        B.perform_async
        C.perform_async
        D.perform_async
        E.perform_async
        F.perform_async
      end
    end
  end

  def step2_done(status, options)
    # implement in a similar fashion to step1_done, kicking off G
  end

  def shipped(status, options)
    oid = options['oid']
    puts "Order #{oid} has shipped!"
  end
end

In this manner, we can implement serial steps in the workflow, with the jobs in each step executing in parallel.