Repository navigation
Replies: 1 comment 9 replies
|
Incrementing delivery count for every return as we did prior to 4.3 was simply incorrect. That is not how it was meant to be used but was instead a side effect of technical limitations. When these were resolved in 4.3 the behaviour was corrected but in doing so we noticed that there was no way to return a message and increment delivery count in the legacy protocol. Hence we re-interpreted basic.reject to fill this role. We felt this was acceptable as poison message handling is a protection mechanism not a feature that a client application makes use of for it's messaging logic. Regarding the consumer timeout not incremented the delivery count I am happy to reconsider this. Originally we were quite strict in only incrementing delivery count when when are pretty sure the processing result was a failure. Arguably a timeout like this should be consider a failure. The earliest we could ship this change would be for RabbitMQ 5.0.
|
Uh oh!
There was an error while loading. Please reload this page.
Community Support Policy
RabbitMQ version used
4.3.x
How is RabbitMQ deployed?
Other
Steps to reproduce the behavior in question
Summary
RabbitMQ 4.3 changed quorum queue poison-message handling so that
delivery-limitis evaluated againstdelivery-countrather thanacquired-count.As part of this change, rabbitmq-server#15387 introduced different delivery-count semantics for the two AMQP 0-9-1 negative acknowledgement methods:
basic.reject(requeue=true)incrementsdelivery-countbasic.nack(requeue=true)does not incrementdelivery-countThis distinction is now described in RabbitMQ's
basic.rejectvs.basic.nackdocumentation and in the RabbitMQ 4.3 quorum queue documentation under When is delivery count incremented?.The current RabbitMQ 4.4 documentation describes the same semantics, so this is not limited to the 4.3 release series.
The same delivery-count table says that a consumer timeout increments
acquired-count, but does not incrementdelivery-count.As a result, existing consumers that use
basic.nackto requeue failed deliveries can change from bounded retries followed by dead-lettering to unlimited redelivery after upgrading to RabbitMQ 4.3 or later.A message that repeatedly causes consumer handlers to become stuck can similarly be returned by the consumer timeout without progressing towards
delivery-limit.Why this appears backwards incompatible
The client code, queue declaration,
delivery-limitpolicy and DLX configuration can all remain unchanged, but the observable failure behaviour changes:basic.nack(requeue=true)eventually reachesdelivery-limitbasic.nack(requeue=true)never reachesdelivery-limitbasic.reject(requeue=true)if the return should count towardsdelivery-limitacquired-count-based limit, repeated consumer timeout and redelivery cycles would eventually reachdelivery-limitdelivery-count, so repeated timeout and redelivery cycles do not reachdelivery-limitThe
nackdocumentation page still describesbasic.nackas a method "that provides all the functionality ofbasic.rejectwhilst also allowing for bulk processing of messages". In both the 4.3 and 4.4 versions of the page, this sentence appears directly above the newbasic.rejectvs.basic.nacksection, which now says the two methods differ for poison-message handling.Client libraries and applications have therefore had little reason to expect a poison-message handling distinction between
basic.nack(multiple=false)andbasic.reject.This has also affected existing integrations:
basic.nacktobasic.rejectto restore its RabbitMQ 4.2 behaviour. That fix shipped in RabbitMQ 4.3.7 and 4.4.0, so in RabbitMQ 4.3.0 to 4.3.6 a message rejected by a Shovel destination is redelivered from a quorum source queue without limit.basic.nacktobasic.rejectin Spring AMQP 4.1.0, with the change backported to the 4.0.x and 3.2.x lines.This means that all existing usages of
basic.nack(requeue=true)that rely ondelivery-limitneed to be identified and potentially updated before upgrading. Without that change, poison messages can enter unlimited, high-rate redelivery loops.Consumer timeout concern
RabbitMQ 4.3 documents consumer timeouts as a mechanism for handling slow or stuck consumers. When an AMQP 0-9-1 consumer times out, the delivery is returned to the queue for redelivery. The consumer is cancelled if the client supports consumer cancel notifications; otherwise its channel is closed with
PRECONDITION_FAILED.However, the delivery-count table states that a consumer timeout increments
acquired-countbut notdelivery-count.It is not clear how the two AMQP 0-9-1 paths map onto the table. The table also states that a client crash or connection loss does increment
delivery-count. When the timeout closes the channel rather than cancelling the consumer, it is unclear whether the return counts as a consumer timeout (not counted) or a channel closure (counted).This appears to create a similar, and potentially more difficult, poison message scenario. A particular message may cause a consumer handler to block, deadlock or otherwise stop acknowledging while the connection remains alive. The consumer timeout returns that message to the queue, but the same message can then be delivered to another consumer and cause it to become stuck as well.
If the timeout does not count towards
delivery-limit, the message can repeatedly time out consumers without ever being dead-lettered.Minimal reproduction
basic.nackDeclare a quorum queue with a small
delivery-limitand a DLX.Consume a message with manual acknowledgements.
Repeatedly send:
Observed behaviour:
delivery-limitand is dead-lettered.acquired-countincreases butdelivery-countdoes not, so the message can be redelivered indefinitely.Replacing
basic.nackwithbasic.reject(requeue=true)restores bounded retry behaviour on RabbitMQ 4.3.Consumer timeout (expected from the documentation, not yet verified)
We have not yet reproduced this case. The steps below describe the scenario we expect, based on the documented delivery-count table:
delivery-limit, a DLX and a short consumer timeout.According to the RabbitMQ 4.3 delivery-count table, each cycle increments
acquired-countbut notdelivery-count. The message would therefore not progress towardsdelivery-limit.Requests
1. Please confirm the intended long-term semantics and migration path.
Please could you confirm that the intended usage from RabbitMQ 4.3 onwards is the distinction described in the current documentation:
basic.reject(requeue=true)means that processing failed, the message should be retried, and the attempt should count towardsdelivery-limitbasic.nack(requeue=true)means that the message should be returned without the attempt counting towardsdelivery-limitIn particular, please could you confirm whether applications that currently use
basic.nack(requeue=true)for failed processing are expected to update those usages tobasic.reject(requeue=true)when they rely on quorum queue poison-message handling?That would be a potentially large migration across existing application and client-library code. Before making that change, we want to confirm that this is the intended long-term semantic distinction, and one that applications and client libraries should align with going forward.
We would like to avoid implementing a broad migration for RabbitMQ 4.3, only to have to pivot again if these semantics change in a later release. Clear documentation and confirmation of the intended long-term usage would help us choose a stable migration path.
Please could you also clarify the intended use case for
basic.nack(requeue=true)when it allows an unsuccessfully processed delivery to be returned an unlimited number of times?In particular, when handling a single delivery, when should an AMQP 0-9-1 application use
basic.nack(requeue=true)rather thanbasic.reject(requeue=true)?2. Please confirm the intended behaviour for consumer timeouts.
The RabbitMQ 4.3 delivery-count table states that a consumer timeout increments
acquired-countbut notdelivery-count.Please could you confirm that it is intentional for repeated consumer timeouts not to count towards
delivery-limit?Please could you also confirm whether this applies to both AMQP 0-9-1 timeout paths: when the consumer is cancelled, and when the channel is closed with
PRECONDITION_FAILEDbecause the client does not support consumer cancel notifications?What is the recommended mechanism for ensuring that a message which repeatedly causes consumer handlers to become stuck is eventually dead-lettered or otherwise quarantined?
Unlike the
basic.nackcase, the consumer cannot switch tobasic.rejectbecause it has become stuck and never gets an opportunity to settle the delivery.Would it be more appropriate for a consumer timeout to count as a failed delivery, or for this behaviour to be configurable?
3. Please clarify whether
basic.rejectcan be used consistently across RabbitMQ versions.There may be several RabbitMQ versions in use during a migration. For code that currently uses
basic.nack(multiple=false, requeue=true)for a single failed delivery, is there any reason not to replace it withbasic.reject(requeue=true)across both older and newer RabbitMQ versions?Using
basic.reject(requeue=true)consistently would simplify the migration, but we would like to confirm that there are no relevant behavioural, compatibility or client-library concerns when making that change against pre-4.3 brokers.4. Please document this prominently as a behavioural compatibility change.
The RabbitMQ 4.3 delivery-count table explains the new behaviour clearly, but applications upgrading from 4.2 may not know that existing
basic.nackusage and consumer timeout behaviour need to be reviewed.#15387 (milestone 4.3.0) is not mentioned in the release notes for any 4.3.x release or for 4.4.0. The 4.3.0 release notes have a "Breaking Changes and Compatibility Notes" section, but it does not mention this change. Instead, the change appears only as a feature: "Quorum queues now allow unlimited explicit message returns" (#13885).
Please could this be added to the 4.3.0 "Breaking Changes and Compatibility Notes" as a behavioural compatibility change? Please could the
nackdocumentation page also be updated, so that it no longer describesbasic.nackas providing all the functionality ofbasic.reject?5. Would RabbitMQ consider providing a compatibility option?
Would RabbitMQ consider providing a compatibility option that preserves the pre-4.3 behaviour during migration?
This could, for example, be a broker configuration, queue argument or policy that allows
delivery-limitto useacquired-count, or that controls whether operations such asbasic.nackand consumer timeout incrementdelivery-count.6. What is the recommended approach for bulk negative acknowledgements?
What is the recommended replacement for
basic.nack(multiple=true)when a consumer needs every delivery in the batch to count as a failed attempt?Sending an individual
basic.rejectfor each delivery appears to be the only AMQP 0-9-1 option.(I was 50-50 on whether or not to make this an issue or discussion, please feel free to move it to an issue if that makes more sense)
All reactions