Sidekiq large batches and avoiding thread starvation

[RobsonKing]

Some times we need to enqueue a large batch of jobs (example: update x on every account)

• we avoid doing this in a single job because that makes for a long running job.
• when we use multiple jobs this can enqueue thousands of jobs. Even though these jobs are individually quick, it can take minutes for the queue to clear.

When the queue gets back up, other jobs can get delayed. What are the strategies people use to mitigate this?

Option 1: create a new queue for lower priority work. Downside, extra cost to run additional queues
Option 2: I was hoping there was some way to use Batches from sidekiq pro to limit the number of threads a batch could use. Example: enqueue this whole batch of jobs, but only use a maximum of 3 threads so we leave threads for other jobs.
Option 3: make one job enqueue the next one. Downside, much more complicated to setup.

Anyone have any recommendations?