[Proposition] A consumer should be able to bind multiple queues

[bastnic]

A consumer is tied to only one queue in the current implementation, which is a "RabbitMqBundle / Thumper" limitation but not a rabbitmq one. I already implemented it based on php-amqplib, but before coding everything in this repo, I just wanted to check if it seems interesting to other people.

The advantage is quite clear for me: I've a lot of "stuff that are not really that important but must be done someday". For that, I like to have "generic consumer" that are coupled with multiple queues and handle message one by one, coming from any of theses queues. It's a way better scalable architecture as I can launch multiple of these generic consumer instead of launching many of each queue.

The one "problem" I have is to be sure that all the queues of a single consumer are connected to the same exchange, and that can be verified on compilation time.

So… this is just documentation for now, to clarified the need and the how. Thanks.

[phillpafford]

Sounds interesting and I think I get it but your use-case is a little confusing to me. Could you go into more detail?

[bastnic]

Imagine you have like a hundred queues, with some that are hugely used and some that are used in some vague scenario. For the big queues, I've a lot of consumers configured with the fair dispatching. So if a queue is increasing too much, I just add more workers.
This ticket concerns the others small queues. I clearly don't want to have n cli workers launched for queues that have a few jobs a day. So I take all the small queues and add them into a single consumer. So for me it's very very easy to add a new queue: it's just a dev thing (declare new queue, append it to an existing consumer, deploy). My ops doesn't have to launch new consumers with supervisord.

I think it's encouraging to create specific queues with specific callbacks. It's permit to better separate code, and a better async workflow.

The test case is something I've in production since almost a year:

Before that patch, as having 10 workers for so little thing seems too much complicated, the code was :

• job send with routing key "object.send".
• 1 queue is listening… executing task1, task2, task3, task3. The task3 fail, it's requeued, task1 and task2 are done again, but task4, 5, 6, 7, 8, 9 and 10 are never run.

Now, I still have one consumer, but with 10 queues.

• job send with routing key "object.send".
• 10 queues are listening
• one consumer consume them all. If one job fail, all the others are still executed, and the failing queue is increasing: she will be cleared later or by a dev manually launching a debug worker on his computer.

I hope it's more clear for you now. The key point is : queues are good, better is to have a lot of them. But launching workers and handling them is a difficult thing to manage.

[videlalvaro]

I understand the idea, but instead of breaking the existing consumers, why not implement a new pattern that handles your case and then we can try to get that merged?

[bastnic]

It is the idea. I'm just trying to explain my idea BEFORE coding it. I usually find it useful.

On the principle, I want to be able to handle queues in two ways:

• same way as now
• an array "queues" per consumer.

Behind the scene, a new class Queue, and a consumer get one or more queue.

My biggest question is how to ensure the exchange.

[videlalvaro]

The idea of this bundle is to implement certain messaging patterns. If something doesn't fit, then maybe there's a need for a new pattern. If that pattern can be easily described with YAML then I'm up for adding it.

[bastnic]

This PR can be closed in favor of #179, by @joelwurtz.