Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

4.0: use Ra checkpoints in rabbit_fifo for sub-linear time recovery of QQs on boot #10487

Draft
wants to merge 9 commits into
base: qq-v4
Choose a base branch
from
Draft

4.0: use Ra checkpoints in rabbit_fifo for sub-linear time recovery of QQs on boot #10487

wants to merge 9 commits into from

Commits on May 1, 2024

  1. Quorum queues v4 scaffolding. 

    Create the new version but not including any changes yet.

fix

QQ: force delete followers after leader has terminated.

Also try a longer sleep for mqtt_shared_SUITE so that the
delete operation stands a chance to time out and move on
to the forced deletion stage.

In some mixed machine version scenarios some followers will never
apply the poison pill command so we may as well force delete them
just in case.

QQ: skip test in amqp_client that cannot pass with mixed machine versions

QQ: remove dead code

Code relating to prior machine versions and state conversions.

formatting / readability

rabbit_fifo_prop_SUITE fixes
    @kjnilsson
    kjnilsson committed May 1, 2024
    Configuration menu
    Copy the full SHA
    ab64b13 View commit details
    Browse the repository at this point in the history

  2. QQ: add v4 ff and new more compact enqueue command. 

    Also update rabbit_fifo_* suites to test more relevant code versions
where applicable.

add ff mock

QQ: always use the updated credit mode format

QQv4: use more compact consumer reference in settle, credit, return

This introudces a new type: consumer_key() which is either the consumer_id
or the raft index the checkout was processed at. If the consumer is
using one of the updated credit spec formats rabbit_fifo will use the
raft index as the primary key for the consumer such that the rabbit
fifo client can then use the more space efficient integer index
instead of the full consumer id in subsequent commands.

There is compatibility code to still accept the consumer id in
settle, return, discard and credit commands but this is slighlyt
slower and of course less space efficient.

The old form will be used in cases where the fifo client may have
already remove the local consumer state (as happens after a cancel).

Lots of test refactorings of the rabbit_fifo_SUITE to begin to use
the new forms.
    @kjnilsson
    kjnilsson committed May 1, 2024
    Configuration menu
    Copy the full SHA
    3425233 View commit details
    Browse the repository at this point in the history

  3. More test refactoring and new API fixes 

    rabbit_fifo_prop_SUITE refactoring and other fixes.

fixss

bzl

bzl

fixes
    @kjnilsson
    kjnilsson committed May 1, 2024
    Configuration menu
    Copy the full SHA
    1271cee View commit details
    Browse the repository at this point in the history

  4. First pass SAC consumer priority implementation. 

    Single active consumers will be activated if they have a higher priority
than the currently active consumer. if the currently active consumer
has pending messages, no further messages will be assigned to the
consumer and the activation of the new consumer will happen once
all pending messages are settled. This is to ensure processing order.

Consumers with the same priority will internally be ordered to
favour those with credit then those that attached first.

QQ: add SAC consumer priority integration tests

Dialyzer fix

QQ: add check for ff in tests
    @kjnilsson
    kjnilsson committed May 1, 2024
    Configuration menu
    Copy the full SHA
    203b671 View commit details
    Browse the repository at this point in the history

  5. QQ: add new consumer cancel option: 'remove' 

    This option immediately removes and returns all messages for a
consumer instead of the softer 'cancel' option which keeps the
consumer around until all pending messages have been either
settled or returned.

This involves a change to the rabbit_queue_type:cancel/5 API
to rabbit_queue_type:cancel/3.
    @kjnilsson
    kjnilsson committed May 1, 2024
    Configuration menu
    Copy the full SHA
    557b02d View commit details
    Browse the repository at this point in the history

  6. Enable 'remove' consumer cancel mode for AMQP

    @kjnilsson
    kjnilsson committed May 1, 2024
    Configuration menu
    Copy the full SHA
    886b655 View commit details
    Browse the repository at this point in the history

  7. rabbit_fifo: Use Ra checkpoints

    @the-mikedavis
    the-mikedavis committed May 1, 2024
    Configuration menu
    Copy the full SHA
    1e4d4c0 View commit details
    Browse the repository at this point in the history

  8. quorum queues: Use a custom interval for checkpoints

    @the-mikedavis
    the-mikedavis committed May 1, 2024
    Configuration menu
    Copy the full SHA
    5aa1a7a View commit details
    Browse the repository at this point in the history

  9. rabbit_fifo_SUITE: List actual effects in ?ASSERT_EFF failure

    @the-mikedavis
    the-mikedavis committed May 1, 2024
    Configuration menu
    Copy the full SHA
    34e0c5b View commit details
    Browse the repository at this point in the history