Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Messenger][AMQP] Set the delayed queue as lazy #35619

Open
wants to merge 1 commit into
base: master
from

Conversation

@lyrixx
Copy link
Member

lyrixx commented Feb 6, 2020

Q A
Branch? master
Bug fix? no
New feature? no
Deprecations? no
Tickets
License MIT
Doc PR

Since RabbitMQ 3.6.0, the broker has the concept of Lazy Queues - queues that move their contents to disk as early as practically possible, and only load them in RAM when requested by consumers, therefore the lazy denomination.

One of the main goals of lazy queues is to be able to support very long queues (many millions of messages). Queues can become very long for various reasons:

  • consumers are offline / have crashed / are down for maintenance
  • there is a sudden message ingress spike, producers are outpacing consumers
  • consumers are slower than normal

By default, queues keep an in-memory cache of messages that is filled up as messages are published into RabbitMQ. The idea of this cache is to be able to deliver messages to consumers as fast as possible. Note that persistent messages can be written to disk as they enter the broker and kept in RAM at the same time.

From: https://www.rabbitmq.com/lazy-queues.html

It's typically the case of delay queue


WARNING: This is a BC Break.

An exception similar to the following will happen if the delayed queue
already exist:

AMQPQueueException {#664
  #message: "Server channel error: 406, message: PRECONDITION_FAILED - inequivalent arg 'x-queue-mode' for queue 'test.load' in vhost 'joli_redirection': received 'lazy' but current is 'default'"

I don't know how to fix that:

  1. We could try to create the queue as lazy and if it does not work, as normal.
  2. We could add an option in the config, and it's up to the user to do what it should be done

Note: It's possible to change the type at runtime

> Since RabbitMQ 3.6.0, the broker has the concept of Lazy Queues - queues that move their contents to disk as early as practically possible, and only load them in RAM when requested by consumers, therefore the lazy denomination.
>
> One of the main goals of lazy queues is to be able to support very long queues (many millions of messages). Queues can become very long for various reasons:
>
> * consumers are offline / have crashed / are down for maintenance
> * there is a sudden message ingress spike, producers are outpacing consumers
> * consumers are slower than normal
>
> By default, queues keep an in-memory cache of messages that is filled up as messages are published into RabbitMQ. The idea of this cache is to be able to deliver messages to consumers as fast as possible. Note that persistent messages can be written to disk as they enter the broker and kept in RAM at the same time.

From: https://www.rabbitmq.com/lazy-queues.html

It's typically the case of lazy queue
@lyrixx lyrixx requested a review from sroze as a code owner Feb 6, 2020
@nicolas-grekas nicolas-grekas added this to the next milestone Feb 6, 2020
Copy link
Member

Tobion left a comment

The PRECONDITION_FAILED error can be solved by either changing the naming structure of delay queues or with a feature flag. Also there is a possiblity to ask rabbitmq if the queue with the arguments already exists which you could use to skip the failing declaration.

@lyrixx

This comment has been minimized.

Copy link
Member Author

lyrixx commented Feb 7, 2020

@Tobion I like naming structure move. (simple, easy)

What's you favorite one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.