Priority queues

[guillemcabrera]

We've been evaluating different queuing alternatives for golang with a Redis backend. Machinery seemed to be promising, but we were worried about different priority queues usage.

We're building a system in which several workers should share a queuing system to pick and perform tasks. We want some tasks to have higher priority than others and workers to pick first from higher priority queues if they contain messages.

From the examples we reviewed, it seems machinery is only capable to work with a single default queue. Is there any way to implement the desired behaviour? In case not, is there any plan to include this at any time?

We had other systems written in Python and we switched from Celery to rq because of the priority management in the former.

Thanks!

[RichardKnop]

@guillemcabrera You could do something "similar", where you can define multiple queues for different kind of tasks. And let's say you could have 4 workers consuming the important queue and 2 workers consuming the less important queue.

This way you could achieve at least something close to what you need, i.e. there would be more computing power for priority tasks.

Just have multiple workers with different config.DefaultQueue defined, one set for priority tasks, and set for less important tasks.

This is as close as you can get to priority queues like you are describing right now.

I am thinking about adding support for RabbitMQ priority queues but currently I am a bit busy with another project so it will have to wait. There is no plan to add this feature in immediate future.

Of course, if you really need it and have decided to use machinery, I am open to PRs. You could implement it yourself.

[RichardKnop]

With AMQP you could also take advantage of different routing algorithms. There is already support for that: https://github.com/RichardKnop/machinery/blob/master/v1/signatures/signatures.go#L27