Simple vs Complex Workflows in Sidekiq Batch: Use Cases and Trade-offs

[zoec0110]

Hi,

I'm currently working on a workflow using Sidekiq Batch. My current implementation is simple, similar to this:

class SimpleWorkflow
  include Sidekiq::Job
  sidekiq_options retry: 3, queue: :default

  def perform
    batch = Sidekiq::Batch.new
    batch.on(:success, 'SimpleWorkflow#step2')
    batch.jobs do
      SimpleStep1Worker.perform_async
      SimpleStep1Worker.perform_async
      SimpleStep1Worker.perform_async
    end
  end

  def step2
    batch = Sidekiq::Batch.new
    batch.on(:success, 'SimpleWorkflow#step3')
    batch.jobs do
      SimpleStep2Worker.perform_async
      SimpleStep2Worker.perform_async
    end
  end

  def step3(status, options)
    batch = Sidekiq::Batch.new
    batch.jobs do
      SimpleStep3Worker.perform_async
    end
  end
end

class SimpleStep1Worker
  include Sidekiq::Job

  def perform
    puts "step 1"
  end
end

class SimpleStep2Worker
  include Sidekiq::Job

  def perform
    puts "step 2"
  end
end

class SimpleStep3Worker
  include Sidekiq::Job

  def perform
    puts "step 3"
  end
end

This approach meets my current needs for sequential task execution, and I found the order of execution is correct. However, I've noticed the "Really Complex Workflows with Batches" pattern in the documentation.

My questions are:

What are the main advantages of using the Complex Workflows approach over a simpler implementation like mine?
In what specific scenarios would the Complex Workflows pattern be necessary or significantly beneficial?
Are there any performance implications or scalability concerns when choosing between these approaches?
For someone starting with Sidekiq Batch, would you recommend beginning with a simpler approach, or is it better to adopt the Complex Workflows pattern from the start?

I notice that there is a parent_bid associated with child batches, but in my case, it doesn't seem necessary to use it.
Thank you!

[mperham]

The only drawback is that you can't run a callback when the entire workflow is done. A success callback on step3 is similar but implicitly assumes it is part of the batch workflow which can cause other issues (like testing it outside of a workflow).

[zoec0110]

You said testing might be tricky if we do it outside the whole workflow. Could you give me an example of what kind of problems we might run into if we try to test step3 by itself?

Thanks a lot for your reply!

[mperham]

Your step3 callback may assume that step1 and step2 have already run and execute some sort of workflow finalizer. Since step3 is running on its own in test, that assumption could be invalid but it's all up to your implementation.