Sidekiq::Limiter when OverLimit does not handle Sidekiq::Batch jobs correctly

[mcheemaa]

Background -

We have a use case where we are creating multiple jobs per row from a csv inside a Sidekiq::Batch and triggering an action in the on_complete callback. We can not use the on_success callback as some jobs can fail with actual errors so we want all jobs to be completed before triggering the callback as the on_success callback does not get triggered even if 1 of 100 jobs fail so if 99 jobs pass and 1 actually failed we want to trigger the on_complete callback to do some action.

Issue -

The issue comes when we have aSidekiq::Limiter.concurrent inside the batch job and it errors out with Sidekiq::Limiter::OverLimit and multiple workers try different jobs once the callback thinks that the job was completed at least once.

However, I am thinking that the job to be retried if it was from a batch and errored out due to the concurrency limit should be added back to the parent batch that the job came from so in reality the on_complete will not get triggered for this Sidekiq::Limiter::OverLimit error and the job will be retried inside the batch - only triggering the callback once the job was actually performed once.

We have thought about maybe creating a custom solution to pass the batch_id, rescue Sidekiq::Limiter::OverLimit error and re enqueue the job inside the existing batch and create an incremental delay logic on our end, but I am just wondering if there is a solution for this already that someone has already found or if sidekiq thinks this will be a possible enhancement to the batches or concurrency limiter.

I can add more details if this above does not make sense or if anyone wants to talk about it more.