Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected PublishConfirmExceptions with 2.x #270

Closed
cocowalla opened this issue Sep 24, 2017 · 7 comments
Closed

Unexpected PublishConfirmExceptions with 2.x #270

cocowalla opened this issue Sep 24, 2017 · 7 comments

Comments

@cocowalla
Copy link

cocowalla commented Sep 24, 2017

I'm seeing some odd behaviour when the broker goes down when using using pub/sub.

After the broker comes back up, every single publish results in an immediate PublishConfirmException with message The broker did not send a publish acknowledgement for message # within 0:00:10.

Messages are still being delivered though.

@pardahlman
Copy link
Owner

pardahlman commented Sep 25, 2017

Thanks for reporting this. I must admit that I haven't tried this myself, but it is something that definitely should work. It is very possible that the publish sequence resets after the connection goes down and recovers. There is always the option to disable publish confirms alltogether by calling you outing operations (publish etc) with .UsePublishAcknowledge(false).

@cocowalla
Copy link
Author

Yes, after the broker restarts I see a single entry in PublishAcknowledgeMiddleware with NextPublishSeqNo set back to 1.

I'd like to keep publish confirmations if at all possible :)

@pardahlman
Copy link
Owner

I've looked into this and adjusted the internal counter for the publish sequence to reset if the sequence has for some reason been reset.

This solution seems to be robust for multiple concurrent publishers using the same channel.

@cocowalla
Copy link
Author

This seems to work, unless any of the publish tasks were cancelled while the broker was down.

For example, if the publish call looks something like this (if the publish doesn't succeed in 5s, cancel the task):

using (var timeoutCts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
{
    await bus.PublishAsync(message, configAction, timeoutCts.Token);
}

When the broker comes back up, every subsequent publish results in an immediate PublishConfirmException.

@pardahlman
Copy link
Owner

Are you sure? I get an OperationCanceledException, which is expected as the task has been cancelled. Personally I think this is odd behavior - but that is how cancellation is handled.

try
{
  await _client.PublishAsync(new ValueCreationFailed(), token: timeoutCts.Token);
}
catch (OperationCanceledException) { }

@cocowalla
Copy link
Author

Gah, my mistake, I was working with the wrong source - your commit has fixed the issue in all cases!

@xwrs
Copy link

xwrs commented Aug 31, 2018

Hi. I have faced the same issue in rc5. With moderate load of ~400 events. Singular publishes work fine. Increasing wait time for publish confirm to 1 minute is not helping. Events are all delivered though. Temporarily disabled publish ack with .UsePublishAcknowledge(false) but it would be nice to understand why it is happening.

"Depth": 0,
"ClassName": "",
"Message": "The broker did not send a publish acknowledgement for message 281 within 0:01:00.",
"Source": "System.Private.CoreLib",
"StackTraceString": " at RawRabbit.Operations.Publish.Middleware.PublishAcknowledgeMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Operations.Publish.Middleware.ReturnCallbackMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Pipe.Middleware.PooledChannelMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Pipe.Middleware.HeaderSerializationMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.Pipe.Middleware.ExchangeDeclareMiddleware.InvokeAsync(IPipeContext context, CancellationToken token)\n at RawRabbit.BusClient.InvokeAsync(Action1 pipeCfg, Action1 contextCfg, CancellationToken token)\n at .......

P.s. @pardahlman thanks for awesome framework

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants