[watermill-amqp] construct subscriptions using a single AMQP connection

[ferbs]

At present, amqp/subscriber.go establishes a new AMQP connection each time a Subscriber is instantiated. RabbitMQ recommends sharing a single connection. The AMQP connection is heavy, consuming ~ 100 KB of RAM each on the RabbitMQ side and slow to establish. (Citation.)

Ideally, Watermill's amqp.NewSubscriber method would automatically reuse a single connection, creating instead a new Channel. If this isn't feasible, please consider adding a new method that accepts an existing *amqp.Connection parameter (vs the current amqp.Config param.)

It's worth mentioning, the above might require updating reconnect logic, to manage the case of concurrent reconnect attempts by multiple subscribers sharing the same connection.

[jawr]

An extension of this would mean we could remove/stop router handlers dynamically. At the moment the Handler shutdown process attempts to close the subscriber, which if shared would be undesired.

[raqbit]

Ideally, Watermill's amqp.NewSubscriber method would automatically reuse a single connection, creating instead a new Channel

The same could apply to publishers as well. For instance I'm working on a "processor" of sorts which acts on messages in one queue and outputs new messages to an output queue via a publisher. Currently watermill would create two connections, one for the subscriber and one for the publisher.

If this isn't feasible, please consider adding a new method that accepts an existing *amqp.Connection parameter (vs the current amqp.Config param.)

In this case you'd lose all benefits of having watermill take care of connection management for you. I think it'd be better if watermill somehow exposed the internal connection manager and allowed us to re-use that instead.

Another thing I noticed while looking at watermill's AMQP package is that an AMQP channel is created for each publish, which also isn't recommended. (Citation.)

In an ideal world the AMQP connection should be created once (unless you need multiple configs), the subscriber channel should be created once (per subscriber) & be long-lived and the same goes for the publisher channel.

[roblaszczak]

Hello, I will be happy to merge any changes that are working and not affecting current functionality and flexibility :)

Feel free to experiment, with current tests It should be pretty simple. If you need any help, please hit me on Gophers slack so I can help. To make it easier, I shared some tribal knowledge about how I do debugging of the tests here: https://watermill.io/docs/troubleshooting/#debugging-pub-sub-tests

[bstasyszyn]

I have a pull request open that (hopefully) resolves this issue: ThreeDotsLabs/watermill-amqp#9. Please review. Thx! @roblaszczak @ferbs @jawr @raqbit

[ferbs]

@bstasyszyn I've switched off the project using watermill-amqp so will go silent (leaving this issue open.)

[bstasyszyn]

PR was merged: ThreeDotsLabs/watermill-amqp#9. This issue can be closed.