Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

RabbitMQ queue bindings are lost after a reconnection #1701

Closed
alan0428a opened this issue Jun 18, 2021 · 2 comments · Fixed by #1716
Closed

RabbitMQ queue bindings are lost after a reconnection #1701

alan0428a opened this issue Jun 18, 2021 · 2 comments · Fixed by #1716
Labels

Comments

@alan0428a
Copy link

Hello there,

I was checking the auto-reconnect mechanism of RabbitMQ. Then I found that after a reconnection to RabbitMQ, the exchange and all queues are restored but the bindings between them are lost.
image

After carefully tracing the code and the RabbitMQ document, I think I might have the idea. The document states that

The following steps are performed for every channel known to being open at the time of connection failure:

  • Re-declare exchanges (except for predefined ones)
  • Re-declare queues
  • Recover all bindings
  • Recover all consumers

It means that if we want the bindings to be recovered after reconnection, channels must be opened at the time of connection failure. But in the DoInternalSubscription method, we use "using" to declare a channel, so the channel is being disposed when leaving the "using" section.

        private void DoInternalSubscription(string eventName)
        {
            var containsKey = _subsManager.HasSubscriptionsForEvent(eventName);
            if (!containsKey)
            {
                if (!_persistentConnection.IsConnected)
                {
                    _persistentConnection.TryConnect();
                }

                using (var channel = _persistentConnection.CreateModel())
                {
                    channel.QueueBind(queue: _queueName,
                                      exchange: BROKER_NAME,
                                      routingKey: eventName);
                }
            }
        }

Maybe we can use the _consumerChannel which is created at the constructor instead, rather than declare a new channel whenever needs to bind a queue to an exchange? What do you think?

The document also states that

Much like connections, channels are meant to be long lived. That is, there is no need to open a channel per operation and doing so would be very inefficient, since opening a channel is a network roundtrip.

So I think the fix might also improve the performance as well?

I have tested the approach and it works without issue. I am happy to create a PR if needed :)

How to reproduce

  1. Launch the project using docker-compose ( I was using visual studio)
  2. Stop the RabbitMQ container
  3. Wait services start trying to reconnect
  4. Start the RabbitMQ container
  5. Go to RabbitMQ admin UI to check bindings
@alan0428a
Copy link
Author

Any thoughts or suggestions on this? Or if I miss something?

@sughosneo
Copy link
Contributor

sughosneo commented Jul 5, 2021

Hi @alan0428a, sorry for the delay in responding. Thank you for identifying the issue and sharing the details repro steps 👍

Yes, your right. If the RabbitMQ container/pod gets restarted, the binding between the exchange and queues were lost. We don't need to create the channel everytime, instead we could use a common _consumerChannel like the connection _persistentConnection If you could submit a PR, we will be happy to review and merge the changes.

Also, in the k8s environment, the same repro steps should fetch similar results, if we delete the RabbitMQ pod. You may want to check that too.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants