Consumer fails to re-bind to queue after connection to rabbitmq was closed/re-established. #72
Comments
|
This looks like a bug with the Java driver. Try using amqp-client 3.6.1 with RabbitMQ and see if it still suffers, or report a bug with them? |
|
Thanks, I will try upgrading and see if the issue persists. I can re-open this issue if it's not solved. |
|
TL;DR: If you have this, or similar problem with disconnects, use consume(Queue.passive(topic(queue("wow-maybe-queue"), List("some-topic.#"))))Full explanation: In case anyone faces similar issues (they still occur with op-rabbit 1.3.0), in a race-condition way (sometimes occurs, sometimes not), some explanations: The exact problem is that after a disconnect to rabbitmq (forced from rabbitmq side by e.g. closing the connection via the rabbitmq web interface), op-rabbit successfully reconnects to rabbitmq, but fails to bind a queue. Start a consumer (all as supposed to be): let's force a connection close via rabbitmq console (this time, the race condition is on our side, reconnect is successful) repeat: let's force a connection close via rabbitmq console: this time we see the error. The result is a consumer that is stuck and never recovers. I was able to reproduce this error about 30% of the time. I tried different versions of the java If you require it, I'm able to write a failing test (though it takes a bit of work as it involves a running mq instance to contact & force close a connection in the test); let me know if you need this. The workaround, which does not exhibit that problem, is to use Queue.passive with a queue definition: consume(Queue.passive(topic(queue("wow-maybe-queue"), List("some-topic.#"))))Perhaps this could be the default behaviour? |
jschaul
changed the title
|
Hello, |
|
Hi @fedragon - Thank you so much for your detailed report. I've read and have considered it. Part of the problem here is that sometimes bindings fail legitimately, and crashing the actor is the appropriate thing to do. However, the case you describe here is clearly an exception. It might be worthy to note that the problem isn't exactly the shared mutable state, although that can be a problem. Doing all channel operations in the channelActor sub-optimal for performance reasons. I am careful to not concurrently apply operations to the channel, and it is always possible for a channel to close right under your feet. Eventually, I would like all channel operations for a consumer to happen in the consumer actor. I pushed up a potential fix for this issue. I look at the channel state after a BindFailure is received. If the channel is marked as closed, which it should be after such an exception, then I ignore the exception and wait for a new channel to be created and sent (depending on the behavior of ChannelActor). Can you please test this and see if it helps your case? If it is possible to write a test for it, it would be really great. However, if the test is going to be brittle due to the complexity of testing for this, it may be okay to forego it. (PS: careful != correct! On review I noticed and fixed a potential issue with |
|
Thank you @timcharper we will test it and come back to you as soon as possible! |
|
Great, thank you! I just pushed |
|
This looks promising, thank you! I'll test and let you know. |
|
Hello @timcharper, I've tested this manually and it works, successfully rebinding to the queue after each disconnect. |
|
\o/ |
|
My manual tests also do not exhibit the faulty behaviour anymore on 1.5.0-RC1. |
UPDATE: see #72 (comment) for explanation/workaround of the problem. Original thread left for reference below.
Either op-rabbit or the underlying libraries seem to cause an unrecoverable crash for op-rabbit based consumers after the rabbitmq instance is unavailable (e.g. crashed itself) for a short period of time. Once the actual RabbitMQ is back up, the expected reconnect does not happen for the consumers. Instead the following errors occurred (see stack traces below)
This happened with version 1.2.1 of op-rabbit.
Is there a way to handle this and try to reconnect?
logger name:
akka.actor.OneForOneStrategyfull stacktrace:
followed by a
logger name:
com.spingo.op_rabbit.SubscriptionActormessage:
An error while trying to bind a consumer to <queue-name>full stacktrace:
The text was updated successfully, but these errors were encountered: