Processing all queues without having to name them explicitly

[stevenharman]

We have a small number of weighted queues defined in our config YAML. Something like this:

:queues:
  - ['within_30_seconds', 5] # 50% of the time we check this queue
  - ['within_5_minutes', 2]  # 20% of the time
  - ['within_1_hour', 2]     # 20% of the time
  - ['within_24_hours', 1]   # 10% of the time

We've also defaulted Sidekiq to one of those queues, so we have no "default" queue. This all works out fine as bin/sidekiq will pick up the config file and watch each of our 4 queues.

In production we have certain machines (Heroku Dynos, but same idea) process certain queues by passing the -q CLI option. This also works out great.

However, we're now wanting to add some special queues that will only be processes by a certain resource class of machines in production. Basically, a subset of our workload demands A LOT of memory. We still want to maintain our latency-based queues, but have a way to push those memory-hungry workloads to queues which will only be processes by the machines with lot o' memory. We're thinking of going with queue names something like within_30_seconds.memory_intensive, within_5_minutes.memory_intensive, etc…

So we'll end up with 8 different queues. 4 of them will run on whatever size machine. And 4 of them ONLY on machines with lots of memory. Using -q with weights on the CLI when launching those sidekiq instances will do just that. Great!

But locally, for developer ergonomics, it'd be really nice to only have to run a single sidekiq instance and have it process all of the queues. Is there a way to give a wildcard (of sorts)? Or should we just define the full set of queues in our sidekiq.yml, knowing that we won't ever use the queues: defined there in production (i.e., that list is our dev ergonomics mechanism)?

Thank you.

[mperham]

There’s no way to do this. I’d recommend a Procfile.dev which lists all your queues.

[stevenharman]

Thanks Mike. I figured as much, but wanted to double check (and get an answer written down for others looking/future me 😀).