Queue escalation?

[cpuguy83]

Is there a way to escalate a job from one queue to another?
The idea of this being that some server is assigned to work on a queue but goes down and is unable to perform those tasks so the job sits there because nothing is available to pick it up, another server would be available to work on those jobs if the job has reached a configurable escalation time.

This may be a job for middleware, not sure.

[betelgeuse]

How about making the other server also work on the queue with for example a lower priority (for example I have configured sidekiq for strict ordering by using a sufficiently large priority)?

[cpuguy83]

Technically yes, and I actually do this on another project, but sometimes you don't want the server to be doing the extra work and if the job queue on that server is empty it's going to pick them up.

[betelgeuse]

Changing the queues of a sidekiq process currently requires a restart so what you want would need to be implemented in external monitoring infrastructure. The first step towards helping your use case would be to allow configuring the listened queues at runtime but @mperham do you think this is a good idea?

[mperham]

I think it's a sufficiently rare use case that it's probably not a good idea. At a high level, you are trying to work around a SPOF. Architect your system so you don't have SPOFs in the first place and these complexities won't be necessary.

[cpuguy83]

Actually the main goal for me personally is to use sidekiq as a job queue for distributed system monitoring checks (nagios replacement) where edge nodes essentially have their own work queue to perform services checks on themselves and if the edge node goes down the main server(s) would take over the roll of checking that edge node (and flag it as down/unknown)... I suppose there are other ways of dealing with it, of course.

BTW, Mike I recently switched to Sidekiq from DJ on a completely separate project and am absolutely loving it.

[mperham]

👍 🍰