Queue process termination and restart causes consumer to stop receiving messages #13369
-
Describe the bugWe are noticing some odd behavior when our queue processes crash. The reason for the crashing is hard to identify, we are assuming some windows security software is interacting poorly with Rabbit. When the queues do crash, our consumers tied to those queues stop receiving messages. When the queues automatically restart the consumers still do not receive messages. The channels used by these consumers do not encounter a ChannelShutdown event, so our software is not detecting that anything has gone wrong with the rabbit process. Snippet of Queue crash stack:
Snippet of Queue Restart The below link seems to be a related issue, and the guidance was to check in on security software. We have done this to the best of our ability by disabling the security software that was exposed to use, but we don't fundamentally control the host, so its possible that other security software is interacting with RabbitMQ. Any guidance on:
Reproduction steps1.Have a consumer with a channel that is bound to Queue A Expected behavior
Additional context
|
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 8 replies
-
|
Versions: |
Beta Was this translation helpful? Give feedback.
-
is the line you are looking for. A filesystem operation has failed with Queue replicas and channels are completely independent from each other, although channels can monitor queues. Channels do retry some operations with a delay when a queue does not have an elected leader (in the case of quorum queues and streams). So for non-transient data, use one of those two. For transient (specifically exclusive, since non-exclusive non-durable queues are going away later in the 4.x series), there already is #12949 which is not as trivial to address as it may sound. |
Beta Was this translation helpful? Give feedback.
-
|
A highly related protocol extension: consumer cancellation notifications. A failure is not the same scenario as a deletion, because in the latter case usually (sadly far from always) a queue with the same name is unlikely to re-appear in the short term. With failures, you never know because that depends entirely on the underlying cause. The |
Beta Was this translation helpful? Give feedback.
-
|
Okay gotcha, thanks for the response. Yeah we had identified this as an issue stemming from a filesystem operation. Resolving why we are getting filesystem errors has been painful thus far. So when using classic queues, this behavior is expected is seems. So to get some clarity, if we moved to using quorum queues, would our channels then be able to tolerate/detect message store index exceptions. Just some added detail, we are not utilizing clustering of our rabbitmq server. We have a single instance of the RabbitMQ Server. Our classic queues are durable and non-exclusive. Thanks again for the swift response! |
Beta Was this translation helpful? Give feedback.
-
|
Also, you mentioned this consumer cancellation extension. Do you think that would this message store index exception and allow us to respond to that? Im pretty fresh to rabbit, so im not following the failure/deletion comment you made. Also you are a saint haha |
Beta Was this translation helpful? Give feedback.
-
|
Consumer cancellation notifications were introduced to allow applications observe queue deletion. I suspect they have never been adopted for quorum queues and streams, which may change as part of #12949. The parts that could have sent them for CMQs are gone together with CMQs in 4.x. |
Beta Was this translation helpful? Give feedback.
-
|
Hi guys, This issue appears to be related to what we previously encountered here. We still experience this problem occasionally, but only on Windows. It seems to be caused by an unknown antivirus blocking our process, leading to consumer crashes with no recovery options. Does anyone know of a reference page listing similar cases and recommended AV exclusions to mitigate this issue? Such a resource could also be useful for troubleshooting similar problems. Thanks in advance! |
Beta Was this translation helpful? Give feedback.
-
Personally I'd recommend removing any and all "security" or antivirus software from Windows servers running RabbitMQ. Otherwise, create exclusions for RabbitMQ's data in |
Beta Was this translation helpful? Give feedback.
-
|
@klettier I don't know what kind of "reference page" you are looking for, there is a documentation section. It is extremely common to see obscure issues on Windwos because of anti-virus software that prevents RabbitMQ from accessing it files, opening network connections, and so on. |
Beta Was this translation helpful? Give feedback.
-
|
I know this may not seem directly related to RMQ, so we don't expect much further discussion on this topic. However, I just wanted to share that whenever this issue occurs, we typically downgrade to RMQ server version |
Beta Was this translation helpful? Give feedback.
-
|
@klettier different versions have different disk I/O and CPU usage patterns for many reasons, that can trigger the anti-virus tool. In any case, there is only one right thing to do here. RabbitMQ 3.12 is out of community support and in a few months will be completely out of support. |
Beta Was this translation helpful? Give feedback.
Personally I'd recommend removing any and all "security" or antivirus software from Windows servers running RabbitMQ.
Otherwise, create exclusions for RabbitMQ's data in
%APPDATA%\RabbitMQas well as for the installation directory for RabbitMQ as well as Erlang.