Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure Service Bus Issue - The operation cannot be performed because the brokered message has already been consumed #947

Closed
andrefaceira opened this issue Jul 25, 2017 · 9 comments
Closed

Azure Service Bus Issue - The operation cannot be performed because the brokered message has already been consumed #947

andrefaceira opened this issue Jul 25, 2017 · 9 comments
Assignees
@phatboyg
Labels
Azure Service Bus Critical

Comments

@andrefaceira
Copy link

andrefaceira commented Jul 25, 2017 •
edited
Loading

I've been getting this exception on a daily basis:

[System.InvalidOperationException] Message: The operation cannot be performed because the brokered message '8b040000ff5a000319d808d4d38a3219' has already been consumed. Please use a new BrokeredMessage instance for the operation

I have dug a bit on the MassTransit codebase and found out that there is a retry policy when sending messages to the azure service bus (MassTransit.AzureServiceBusTransport.Transport.TopicSendClient).
One of the retried exception is the TimeoutException which looking at the documentation (https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-exceptions) does NOT guarantee that the process actually failed and the message was not delivered.
So, in the case we get this exception but the message was actually delivered, this brokered message already consumed would make sense, since the message was delivered the first time.

I'm not sure this is the origin of the issue, but that's my only guess at the moment.
A quick fix would be to force the new BrokeredMessages on each retry, but I'm not sure if that would be the best solution for the current design.
EDIT: Having another look at this: we can't blindly new BrokeredMessage again otherwise we may be sending the same message more than once. Not really sure how to proceed here. On our end, we're ignoring this exception since we know the message was delivered, but we're just hiding some issue.

Please let me know what you think or if someone has already had this issue.
Anyway, I'm happy to fix it on a PR if we agree on a solution.
@sharpjs
Copy link
Contributor

sharpjs commented Sep 11, 2017

Ran into this as well. From my non-expert point of view, @andrefaceira's solution seems appropriate: catch and ignore the exception. If the message actually got delivered, that counts as success IMO.

Here's a stack trace of the exception (MassTransit 3.5.6):

System.AggregateException:
System.InvalidOperationException:
   at Microsoft.ServiceBus.Messaging.MessagingUtilities.ValidateAndSetConsumedMessages (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.ServiceBus.Messaging.MessageSender.BeginSend (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.ServiceBus.Messaging.MessageSender.BeginSend (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.ServiceBus.Messaging.MessageSender+<>c.<SendAsync>b__40_0 (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.TaskFactory.FromAsync (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Threading.Tasks.TaskFactory.FromAsync (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.ServiceBus.Common.TaskHelpers.FromAsync (Microsoft.ServiceBus, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at MassTransit.Policies.PipeRetryExtensions+<Attempt>d__3`1.MoveNext (MassTransit, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at MassTransit.Policies.PipeRetryExtensions+<Retry>d__1.MoveNext (MassTransit, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at MassTransit.AzureServiceBusTransport.Transport.ServiceBusSendTransport+<MassTransit-Transports-ISendTransport-Send>d__5`1.MoveNext (MassTransit.AzureServiceBusTransport, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at MassTransit.AzureServiceBusTransport.Transport.ServiceBusSendTransport+<MassTransit-Transports-ISendTransport-Send>d__5`1.MoveNext (MassTransit.AzureServiceBusTransport, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at MassTransit.Transports.PublishEndpoint+<MassTransit-IPublishEndpoint-Publish>d__7`1.MoveNext (MassTransit, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa)
   at MassTransit.Transports.PublishEndpoint+<MassTransit-IPublishEndpoint-Publish>d__7`1.MoveNext (MassTransit, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b8e0e9f2f1e657fa)

@phatboyg
Copy link
Member

Seems like the retry policy should filter out this exception.

@phatboyg
Copy link
Member

Added a try/catch on this one, hope it helps.

@phatboyg phatboyg self-assigned this Dec 22, 2017
phatboyg added a commit that referenced this issue Dec 23, 2017
@phatboyg
Addressed #947, filtering exception when the message was sent but a t…
7b36331 
…imeout exception may have occurred
@pedropaulovc
Copy link

I was wondering, is there a way to backport this fix to MassTransit v3.x using some public extensibility point? Maybe implement a custom IRetryConfigurator?

@phatboyg
Copy link
Member

I'm no longer updating 3.x.

@pedropaulovc
Copy link

Sorry, let me rephrase. I'm not looking for a new MT v3 release. Instead, I'm trying to implement the same fix from 7b36331 using public extensibility point provided by MassTransit. Thanks!

@phatboyg
Copy link
Member

I see what you're asking now. I'm not sure the transport code is pretty low level, it would be hard to cut into it.

@mebjas
Copy link

mebjas commented Oct 9, 2018

@phatboyg Did you find a good solution for this?

@phatboyg
Copy link
Member

phatboyg commented Oct 9, 2018

@mebjas yes, use something newer than 3.5.7...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phatboyg phatboyg

Labels
Azure Service Bus Critical
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@phatboyg @pedropaulovc @sharpjs @andrefaceira @mebjas