QQ: introduce new machine version (2) #3121
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kjnilsson
force-pushed
the
quorum-queues-v2
branch
2 times, most recently
from
f9b2b8a
to
e804229
Compare
ansd
force-pushed
the
quorum-queues-v2
branch
5 times, most recently
from
15c633a
to
5ebc7f5
Compare
ansd
force-pushed
the
quorum-queues-v2
branch
7 times, most recently
from
9dddc0f
to
1b2898e
Compare
ansd
force-pushed
the
quorum-queues-v2
branch
3 times, most recently
from
329ba51
to
1239d28
Compare
ansd
force-pushed
the
quorum-queues-v2
branch
8 times, most recently
from
dbfacc0
to
a7aaa5c
Compare
ansd
force-pushed
the
quorum-queues-v2
branch
2 times, most recently
from
52ce0cf
to
d15bf6d
Compare
|
https://github.com/rabbitmq/rabbitmq-server/tree/qq-resend-protocol-refactor Some work around repeated requeues and resend protocol is in this branch and will be integrated to this branch in time. |
Save per-message memory for dead-lettered messages by
1. removing all headers that are not needed anymore once the message
got dead-lettered (delivery_count and expiry), and
2. saving 1 byte for every lqueue element of the discards queue by using
an improper list instead of tuple to store {Reason, Message}.
In total, this will save 9 bytes per dead-lettered message:
```
1> erts_debug:size({expired, #{delivery_count => 2, size => 3}}).
11
2> erts_debug:size([expired | 3]).
2
```
Many messages can pile up in the discards queue if target queues do not
confirm messages fast enough or not at all.
instead of ?INDEX_MSG and ?DISK_MSG macros because from now on there will be only disk messages. This makes the code more readable.
Decrease memory usage by 6 bytes per message if per-message or per-queue
message TTL is set:
1> erts_debug:size(#{size => 1, expiry => 1000}).
8
2> erts_debug:size([1|1000]).
2
While we could always use a record for the header,
we do want the flexibility of a map.
ansd
force-pushed
the
quorum-queues-v2
branch
2 times, most recently
from
3b4d8a3
to
c44ea79
Compare
ansd
approved these changes
Feb 28, 2022
since they do not exist anymore
because at-least-once dead-lettered messages are arguably consumer messages which should not be purged. According to the AMQP 0.9.1 purge spec: "This method removes all messages from a queue which are not awaiting acknowledgment." At-least-once dead-lettered messages are awaiting acks from target queues.
1. If all messages in the queue are either in 'messages' or 'returns', (i.e. no message currently checked out to consumers) we can directly empty the index. 2. Even if there are consumers, there is no need to delete indexes from the 'messages' queue because we only append to the index once we move a message out of the 'messages' queue. 3. Use lqueue:fold/3 instead of lists:foldl/3 and lqueue:to_list/1
1. A Raft index in ?MSG(RaftIndex, Header) can be 'undefined' if it was a prefix message in v1. Therefore do not use is_integer(RaftIndex) guards. Instead, expect and handle RaftIndex=undefined correctly. 2. The total number of messages did not include any checked out messages after the conversion.
because at-least-once dead-lettering and therefore process rabbit_fifo_dlx_sup does not exist in 3.9. Once the new base version in mixed version tests becomes 3.10 (base version is 3.9 as of March 2022), we can revert this commit.
ansd
force-pushed
the
quorum-queues-v2
branch
4 times, most recently
from
68f1217
to
7aa7453
Compare
|
Docs in rabbitmq/rabbitmq-website#1299 |
Before this commit, when a quorum queue has only expired messages, but the
timer hasn't expired them yet, and a basic.get tries to dequeue a message
the following error occurred:
{{badmatch,
{nochange,true,
{rabbit_fifo, ...},
[{mod_call,rabbit_global_counters,
messages_dead_lettered,
[expired,rabbit_quorum_queue,disabled,1]}]}},
[{rabbit_fifo,apply,3,[{file,"rabbit_fifo.erl"},{line,387}]},
michaelklishin
added a commit
that referenced
this pull request
Mar 8, 2022
QQ: introduce new machine version (2) (backport #3121)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[ ] Publisher de-duplication[ ] Make more compact representations of common commands to reduce disk use