New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redelivery count for messages #502

Open
hairyhum opened this Issue Dec 16, 2015 · 24 comments

Comments

Projects
None yet
@hairyhum
Copy link
Contributor

hairyhum commented Dec 16, 2015

From the 0-9-1 spec:

The server SHOULD track the number of times a message has been delivered to clients and
when a message is redelivered a certain number of times - e.g. 5 times -
without being acknowledged, the server SHOULD consider the message
to be unprocessable (possibly causing client applications to abort), and [dead letter] the message.

This should be made optional for compatibility.

More context is provided in #1020.
Related: rabbitmq/rabbitmq-stomp#36.

@michaelklishin

This comment has been minimized.

Copy link
Member

michaelklishin commented Dec 16, 2015

This is not really a requirement, and message properties don't have a field for redelivery count, only a boolean flag. Still worth having, of course.

@hairyhum

This comment has been minimized.

Copy link
Contributor

hairyhum commented Dec 21, 2015

First investigation results:
Required changes in backing_queue API to support integer redelivery count instead of flag in fetch/2 return value and functions working with fetch.
Also queue index format should be changed to support multiple deliver records, or delivery count. But because current implementation assumes exactly 1 deliver for each acknowledge there should be significant changes in index structure.
Any thoughts?

@vgkiziakis

This comment has been minimized.

Copy link

vgkiziakis commented Mar 1, 2016

Also very interested in this feature. It's invaluable in scenarios where there is a "poisonous" message that for some reason is causing consumers to crash or behave erratically.

Are there any plans for adding this in a future release ?

@hairyhum

This comment has been minimized.

Copy link
Contributor

hairyhum commented Mar 1, 2016

It's in the roadmap for 3.7.x

@mikljohansson

This comment has been minimized.

Copy link

mikljohansson commented Aug 22, 2016

@hairyhum do you know if the delivery-count feature is still slated for inclusion in rabbitmq-server 3.7?

The reason I'm asking is because we see the poisonous-message problem every once in a while. Latest incident was last week and caused a system degradation in our own components, before the problem could be found and circumvented.

We'd very much like a robust broker-supported way of detecting messages that consistently trigger a severe problem in the client

@michaelklishin

This comment has been minimized.

Copy link
Member

michaelklishin commented Aug 22, 2016

@mikljohansson we will do our best to fit it into 3.7.0 but no promises.

@plechev

This comment has been minimized.

Copy link

plechev commented Sep 29, 2016

This is a much needed feature to enhance the robustness of message (re)delivery mechanism. We are dealing with the "poisonous messages" problem which is currently handled by the client, but it will be more elegant if we can use the broker for this.
Thumbs up!

@michaelklishin

This comment has been minimized.

Copy link
Member

michaelklishin commented Nov 2, 2016

An update: this won't be in 3.7.0 because we had quite a few out-of-cycle and new features in 3.7.0 and we need to limit the scope at some point. But we are contemplating it for 3.8.0 (so that it doesn't have to wait for 4.0, that is).

@mattheworiordan

This comment has been minimized.

Copy link

mattheworiordan commented Nov 2, 2016

An update: this won't be in 3.7.0 because we had quite a few out-of-cycle and new features in 3.7.0 and we need to limit the scope at some point. But we are contemplating it for 3.8.0

@michaelklishin out of interest, how do your minor release cycles work i.e. realistically if it made it into 3.8.0, when could we expect that release?

Separately, is there anything we could do to help if it was a priority for us?

@ghost

This comment has been minimized.

Copy link

ghost commented Nov 10, 2016

I posted this issue here as a suggestion for the team to work on it. Your product is really great and I think it would become bigger if retry capabilities are present. Thank you!

@mattheworiordan

This comment has been minimized.

Copy link

mattheworiordan commented Feb 23, 2017

Hi @michaelklishin, do you mind if I ask if any progress has been made on this? Running RabbitMQ on behalf of customers means RabbitMQ is quite susceptible to overload from one customer (unless we rate limit them which is not ideal) simply because of a faulty code loop.

@michaelklishin

This comment has been minimized.

Copy link
Member

michaelklishin commented Feb 23, 2017

@mattheworiordan

This comment has been minimized.

Copy link

mattheworiordan commented Feb 24, 2017

It is on the roadmap for 3.8.0.

Great to hear, thanks Michael.

@rrmayer

This comment has been minimized.

Copy link

rrmayer commented Jan 30, 2018

Do we know what the status is on this issue? We continue to see questions about this on Stack Overflow, so it appears that there is at least a reasonable demand for this feature.

@michaelklishin

This comment has been minimized.

Copy link
Member

michaelklishin commented Jan 30, 2018

The status is that it has a chance of getting into 3.8.0 but we make no promises.

@mattheworiordan

This comment has been minimized.

Copy link

mattheworiordan commented May 16, 2018

The status is that it has a chance of getting into 3.8.0 but we make no promises.

🙏

@jlcd

This comment has been minimized.

Copy link

jlcd commented May 23, 2018

Also interested. This feature would be a life saver, as we won't have to create workarounds to stop requeueing messages that aren't processable.
Any updates whether it will be included into 3.8.0?

@Misiu

This comment has been minimized.

Copy link

Misiu commented Oct 4, 2018

I'm really new to RabbitMQTT but I also think this would be a lifesaver.
Normally when querying external API and getting an exception I'd like to try to make a request for same data couple of times. I'm aware I have Redelivered flag, but I would like to know how many times the same message was redelivered. This way I could reject it when I hit 5 tries. This could be even done on the server.

I have one additional question. Would it be possible to delay redelivery?
I want the same message to be redelivered after a specified time, for example, I want second try after 10 seconds, third after a minute.
Can this be done without publishing a new message?
In C# I can use BasicReject method, but this allows me to flag message to redelivery, but not to delay it.
Can https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/ be used somehow to accomplish this?

@Soarc

This comment has been minimized.

Copy link

Soarc commented Oct 4, 2018

@Misiu to achieve this, i ended up with using MassTransit over RabbitMQ. Something like this

            _bus = Bus.Factory.CreateUsingRabbitMq(sbc =>
            {
                var host = sbc.Host(new Uri($@"rabbitmq://{_host}/"), h =>
                {
                    h.Username(_userName);
                    h.Password(_password);
                });
                sbc.ExchangeType = "direct";
                sbc.AutoDelete = false;
                sbc.Durable = true;
                sbc.Exclusive = false;
                                 
                sbc.ConfigurePublish(PublishCallBack);
                sbc.ReceiveEndpoint(host, _queueName, ep =>
                {
                    foreach (var suber in _handlerSubscribators)
                        suber(ep);                    
                });
                sbc.UseRetry(retryConfig => retryConfig.Immediate(_retryCount)); // <----- This line achives what you want
            });
@lukebakken

This comment has been minimized.

Copy link
Member

lukebakken commented Oct 4, 2018

@Misiu @Soarc - please use the mailing list for discussions.

@Avivsalem

This comment has been minimized.

Copy link

Avivsalem commented Jan 20, 2019

@mattheworiordan,
Did it make the cut for 3.8.0?

@mattheworiordan

This comment has been minimized.

Copy link

mattheworiordan commented Jan 20, 2019

@Avivsalem, I think you were meaning to ask @michaelklishin?

@Avivsalem

This comment has been minimized.

Copy link

Avivsalem commented Jan 20, 2019

Yup... My mistake. So @michaelklishin what do you say? Did it make it to 3.8.0?
Because in my production environment, if a message causes a consumer crash, it will be poison forever...

@rabbitmq rabbitmq locked and limited conversation to collaborators Jan 21, 2019

@michaelklishin

This comment has been minimized.

Copy link
Member

michaelklishin commented Jan 21, 2019

A comment above asks all discussions and questions to be moved to the mailing list.

Quorum queues in master keep track of a redelivery count. We make no promises if classic queues will in 3.8. Currently, it's just a number consumers can use to make decisions, nothing else.

It's your responsibility to handle exceptions in your consumers. This is not going to change with any redelivery count implementation.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.