Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Versioned delivery tags, à la Bunny and Java client #200

Closed
harishvadali opened this issue Jun 9, 2016 · 10 comments
Closed

Versioned delivery tags, à la Bunny and Java client #200

harishvadali opened this issue Jun 9, 2016 · 10 comments

Comments

@harishvadali
Copy link

Hi,
I am testing a recovery scenario by dropping the connection through management console. Below is my scenario.

  1. Put 10 messages in a queue.
  2. Once messages are consumed by the application, wait for 30 seconds and then call BasicAck on the delivery tag received.
  3. While waiting for 30 seconds, I dropped the connection from management console. I noticed that all the Unacked messages are now in Ready state once the connection is back up. For this to happen, I set AutomaticRecoveryEnabled and TopologyRecoveryEnabled to true.
  4. After the Reconnection, my application those messages again with new delivery tags.
  5. Now, for the first set (before reconnection) of 10 messages wait time is up and do BasicAck. Now, I get Channel Exception and that is not getting recovered. At this point, my application is not able to consume or Ack because the channel is down.

After reading the RabbitMq docs, I noticed that the driver is supposed to map the old delivery tags to new delivery tags and ignore any stale delivery tags.

Manual Acknowledgements and Automatic Recovery
When manual acknowledgements are used, it is possible that network connection to RabbitMQ node fails
between message delivery and acknowledgement. After connection recovery, RabbitMQ will reset
delivery tags on all channels. This means that basic.ack, basic.nack, and basic.reject with old delivery
tags will cause a channel exception. To avoid this, RabbitMQ Java client keeps track of and updates
delivery tags to make them monotonically growing between recoveries. Channel.basicAck,
Channel.basicNack, and Channel.basicReject then translate adjusted delivery tags into those used by
RabbitMQ. Acknowledgements with stale delivery tags will not be sent. Applications that use manual
acknowledgements and automatic recovery must be capable of handling redeliveries.

Is this true for .NET driver too? Or Am i missing any other configuration step?

@michaelklishin
Copy link
Member

Please post questions to rabbitmq-users or Stack Overflow. RabbitMQ uses GitHub issues for specific actionable items engineers can work on, not questions. Thank you.

@michaelklishin
Copy link
Member

What you describe is known in other clients as versioned delivery tags. After connection recovery all channels are brand new and have delivery tag counter reset by RabbitMQ. This client does not support them but it should be fairly trivial to add because Java and Ruby clients do.

@michaelklishin michaelklishin reopened this Jun 9, 2016
@michaelklishin
Copy link
Member

Here's a commit that demonstrates how Bunny does it: ruby-amqp/bunny@21e39b9.

@michaelklishin michaelklishin changed the title Channel Exception with Unknown delivery tag Versioned delivery tags, à la Bunny and Java client Jun 9, 2016
@harishvadali
Copy link
Author

@michaelklishin Thanks for the quick response.

@Entroper
Copy link

Entroper commented Jun 9, 2016

We were confused because the .NET documentation says the same thing: https://www.rabbitmq.com/dotnet-api-guide.html#connection-recovery

IModel.BasicAck, IModel.BasicNack, and IModel.BasicReject then translate adjusted delivery tags into those used by RabbitMQ. Acknowledgements with stale delivery tags will not be sent.

Emphasis mine.

I had a look at the code, and I think I've discovered what's happening. The RecoveryAwareModel is responsible for adjusting the tags after a reconnect, and it does have a check to see if the tag is stale before acking. But we were still seeing failed acks in the Rabbit logs, with delivery tags very close to MAX_ULONG. I think this is the root of the problem.

We're testing with the following change, and it appears to be working as expected: Entroper@f1d5bd1

@michaelklishin
Copy link
Member

@Entroper ah, perfect. Recovery in this client is a pretty faithful port of what we have in the Java client, so I expected that delivery tags should be versioned but looked at a couple of places where it wasn't obvious.

Please submit a PR against the stable branch?

@Entroper
Copy link

Entroper commented Jun 9, 2016

@michaelklishin Will do. :)

@Entroper
Copy link

Entroper commented Jun 9, 2016

Fixed in #201.

@michaelklishin
Copy link
Member

Thank you, we will QA it in the upcoming days.

On Thu, Jun 9, 2016 at 11:22 PM, Entroper notifications@github.com wrote:

See #201 #201.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#200 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAAEQoG_HW_D_ihxouHFlxkcOrV61tulks5qKHXwgaJpZM4Ix_jV
.

MK

Staff Software Engineer, Pivotal/RabbitMQ

@kjnilsson
Copy link
Contributor

ok apart from finding a bug related to BasicGet this should now be fixed in stable.

@michaelklishin michaelklishin added this to the 3.6.3 milestone Jun 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants