Refine and fix AMQP 1.0 modified outcome behaviour #6292

kjnilsson · 2022-11-01T11:24:24Z

The commit in f15e2e4 that effectively maps the modified outcome to accepted is not correct. Modified is a lot more like the released outcome. This commit refines the behaviour of the modified in the following ways:

A modified outcome with delivery_failed=true, undeliverable_here=false|undefined will be rejected with requeue=true;
Any other modified outcome will be rejected with requeue=false.

It is worth noting that this isn't completely correct but probably the closest approximation we can achieve at the moment.

The undeliverable_here field refers to the current link, not the message itself but as we have no means of filtering which messages get assigned to which consumer links we instead avoid requeuing it.

Also the case where delivery_failed=false|undefined requires the release of the message without incrementing the delivery_count. Again this is not something that our queues are able to do so again we have to reject without requeue.

This PR also adds API in amqp10_client for sending more advanced modified outcomes.

References #6121 (not yet shipped in a release)

The commit in f15e2e4 that effectively maps the modified outcome to accepted is not correct. Modified is a lot more like the released outcome. This commit refines the behaviour of the modified in the following ways: 1. A modified outcome with delivery_failed=true, undeliverable_here=false|undefined will be rejected with requeue=true; 2. Any other modified outcome will be rejected with requeue=false. It is worth noting that this isn't completely correct but probably the closest approximation we can achieve at the moment. The undeliverable_here field refers to the current link, not the message itself but as we have no means of filtering which messages get assigned to which consumer links we instead avoid requeuing it. Also the case where delivery_failed=false|undefined requires the release of the message _without_ incrementing the delivery_count. Again this is not something that our queues are able to do so again we have to reject without requeue.

Refine and fix AMQP 1.0 modified outcome behaviour (backport #6292)

Refine and fix AMQP 1.0 modified outcome behaviour (backport #6292) (backport #6299)

With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from #6292 (comment) doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true.

This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from #6292 (comment) doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in #10637 (comment) 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com>

This commit contains the following new quorum queue features: * Fair share high/low priorities * SAC consumers honour consumer priorities * Credited consumer refactoring to meet AMQP requirements. * Use checkpoints feature to reduce memory use for queues with long backlogs * Consumer cancel option that immediately removes consumer and returns all pending messages. * More compact commands of the most common commands such as enqueue, settle and credit * Correctly track the delivery-count to be compatible with the AMQP spec * Support the "modified" AMQP 1.0 outcome better. Commits: * Quorum queues v4 scaffolding. Create the new version but not including any changes yet. QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. rabbit_fifo_prop_SUITE fixes * QQ: add v4 ff and new more compact enqueue command. Also update rabbit_fifo_* suites to test more relevant code versions where applicable. QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms. * More test refactoring and new API fixes rabbit_fifo_prop_SUITE refactoring and other fixes. * First pass SAC consumer priority implementation. Single active consumers will be activated if they have a higher priority than the currently active consumer. if the currently active consumer has pending messages, no further messages will be assigned to the consumer and the activation of the new consumer will happen once all pending messages are settled. This is to ensure processing order. Consumers with the same priority will internally be ordered to favour those with credit then those that attached first. QQ: add SAC consumer priority integration tests QQ: add check for ff in tests * QQ: add new consumer cancel option: 'remove' This option immediately removes and returns all messages for a consumer instead of the softer 'cancel' option which keeps the consumer around until all pending messages have been either settled or returned. This involves a change to the rabbit_queue_type:cancel/5 API to rabbit_queue_type:cancel/3. * QQ: capture checked out time for each consumer message. This will form the basis for queue initiated consumer timeouts. * QQ: Refactor to use the new ra_machine:handle_aux/5 API Instead of the old ra_machine:handle_aux/6 callback. * QQ hi/lo priority queue * QQ: Avoid using mc:size/1 inside rabbit_fifo As we dont want to depend on external functions for things that may change the state of the queue. * QQ bug fix: Maintain order when returning multiple Prior to this commit, quorum queues requeued messages in an undefined order, which is wrong. This commit fixes this bug and requeues messages always in the order as nacked / rejected / released by the client. We ensure that order of requeues is deterministic from the client's point of view and doesn't depend on whether the quorum queue soft limit was exceeded temporarily. So, even when rabbit_fifo_client batches requeues, the order as nacked by the client is still maintained. * Simplify * Add rabbit_quorum_queue:file_handle* functions back. For backwards compat. * dialyzer fix * dynamic_qq_SUITE: avoid mixed versions failure. * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * Use QQ consumer removal when AMQP client detaches This enables us to unskip some AMQP tests. * Use AMQP address v2 in fsharp-tests * QQ: track number of requeues for message. To be able to calculate the correct value for the AMQP delivery_count header we need to be able to distinguish between messages that were "released" or returned in QQ speak and those that were returned due to errors such as channel termination. This commit implement such tracking as well as the calculation of a new mc annotations `delivery_count` that AMQP makes use of to set the header value accordingly. * rabbit_fifo: Use Ra checkpoints * quorum queues: Use a custom interval for checkpoints * rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure * QQ: Checkpoints modifications * fixes * QQ: emit release cursors on tick for followers and leaders else followers could end up holding on to segments a bit longer after traffic stops. * Support draining a QQ SAC waiting consumer By issuing drain=true, the client says "either send a transfer or a flow frame". Since there are no messages to send to an inactive consumer, the sending queue should advance the delivery-count consuming all link-credit and send a credit_reply with drain=true to the session proc which causes the session proc to send a flow frame to the client. * Extract applying #credit{} cmd into 2 functions This commit is only refactoring and doesn't change any behaviour. * Fix default priority level Prior to this commit, when a message didn't have a priority level set, it got enqueued as high prio. This is wrong because the default priority is 4 and "for example, if 2 distinct priorities are implemented, then levels 0 to 4 are equivalent, and levels 5 to 9 are equivalent and levels 4 and 5 are distinct." Hence, by default a message without priority set, must be enqueued as low prio. * bazel run gazelle * Avoid deprecated time unit * Fix aux_test * Delete dead code * Fix rabbit_fifo_q:get_lowest_index/1 * Delete unused normalize functions * Generate less garbage * Add integration test for QQ SAC with consumer priority * Improve readability * Change modified outcome behaviour With the new quorum queue v4 improvements where a requeue counter was added in addition to the quorum queue delivery counter, the following sentence from #6292 (comment) doesn't apply anymore: > Also the case where delivery_failed=false|undefined requires the release of the > message without incrementing the delivery_count. Again this is not something > that our queues are able to do so again we have to reject without requeue. Therefore, we simplify the modified outcome behaviour: RabbitMQ will from now on only discard the message if the modified's undeliverable-here field is true. * Introduce single feature flag rabbitmq_4.0.0 ## What? Merge all feature flags introduced in RabbitMQ 4.0.0 into a single feature flag called rabbitmq_4.0.0. ## Why? 1. This fixes the crash in #10637 (comment) 2. It's better user experience. * QQ: expose priority metrics in UI * Enable skipped test after rebasing onto main * QQ: add new command "modify" to better handle AMQP modified outcomes. This new command can be used to annotate returned or rejected messages. This commit also retains the delivery-count across dead letter boundaries such that the AMQP header delivery-count field can now include _all_ failed deliver attempts since the message was originally received. Internally the quorum queue has moved it's delivery_count header to only track the AMQP protocol delivery attempts and now introduces a new acquired_count to track all message acquisitions by consumers. * Type tweaks and naming * Add test for modified outcome with classic queue * Add test routing on message-annotations in modified outcome * Skip tests in mixed version tests Skip tests in mixed version tests because feature flag rabbitmq_4.0.0 is needed for the new #modify{} Ra command being sent to quorum queues. --------- Co-authored-by: David Ansari <david.ansari@gmx.de> Co-authored-by: Michael Davis <mcarsondavis@gmail.com> (cherry picked from commit 194d4ba)

kjnilsson added backport-v3.10.x backport-v3.11.x rabbitmq-amqp1.0 rabbitmq-amqp1.0-client labels Nov 1, 2022

michaelklishin modified the milestone: 3.11.3 Nov 1, 2022

michaelklishin approved these changes Nov 1, 2022

View reviewed changes

michaelklishin merged commit 0e44957 into main Nov 1, 2022

michaelklishin deleted the amqp10-modified-corrections branch November 1, 2022 13:22

mergify bot mentioned this pull request Nov 1, 2022

Refine and fix AMQP 1.0 modified outcome behaviour (backport #6292) #6299

Merged

michaelklishin added a commit that referenced this pull request Nov 1, 2022

Merge pull request #6299 from rabbitmq/mergify/bp/v3.11.x/pr-6292

16159b7

Refine and fix AMQP 1.0 modified outcome behaviour (backport #6292)

michaelklishin added a commit that referenced this pull request Nov 1, 2022

Merge pull request #6302 from rabbitmq/mergify/bp/v3.10.x/pr-6299

b594070

Refine and fix AMQP 1.0 modified outcome behaviour (backport #6292) (backport #6299)

ansd mentioned this pull request Jul 17, 2024

Fix unupported modified outcome behaviour #11730

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refine and fix AMQP 1.0 modified outcome behaviour #6292

Refine and fix AMQP 1.0 modified outcome behaviour #6292

kjnilsson commented Nov 1, 2022 •

edited by michaelklishin

Loading

Refine and fix AMQP 1.0 modified outcome behaviour #6292

Refine and fix AMQP 1.0 modified outcome behaviour #6292

Conversation

kjnilsson commented Nov 1, 2022 • edited by michaelklishin Loading

kjnilsson commented Nov 1, 2022 •

edited by michaelklishin

Loading