Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle messages retrieval failure #324

Merged

Conversation

@stubbst
Copy link
Contributor

commented Apr 19, 2019

When the IStorageConnection service is down, connection.GetPublishedMessagesOfNeedRetry() and connection.GetReceivedMessagesOfNeedRetry() both throw an exception.

The wait operation in ProcessAsync is therefore never executed.

We therefore end up in an infinite loop of immediate retries until the IStorageConnection service is back up. This can be seen in the logs where we have hundreds of "Processor 'DotNetCore.CAP.Processor.NeedRetryMessageProcessor' failed. Retrying..." entries per second.

We add a try/catch around the message retrieval operation and log the error. The next messages retrieval will therefore be executed options.FailedRetryInterval seconds later.

Handle messages retrieval failure
When the `IStorageConnection` service is down, `connection.GetPublishedMessagesOfNeedRetry()` and `connection.GetReceivedMessagesOfNeedRetry()` both throw an exception.

The wait operation in `ProcessAsync` is therefore never executed.

We therefore end up in an infinite loop of immediate retries until the `IStorageConnection` service is back up. This can be seen in the logs where we have hundreds of "Processor 'DotNetCore.CAP.Processor.NeedRetryMessageProcessor' failed. Retrying..." entries per second.

We add a try/catch around the message retrieval operation and log the error. The next messages retrieval will therefore be executed `options.FailedRetryInterval` seconds later.
@yang-xiaodong

This comment has been minimized.

Copy link
Member

commented Apr 20, 2019

Thanks for your PR

It seems that you have not finished yet, I can't find where to call GetSafelyAsync

@yang-xiaodong yang-xiaodong self-requested a review Apr 20, 2019

@stubbst

This comment has been minimized.

Copy link
Contributor Author

commented Apr 20, 2019

Hi. Sorry about that. I forgot to retake two crucial lines from the modifications I made in my project where I overrode some of the classes 😕.

@yang-xiaodong
Copy link
Member

left a comment

LGTM

@yang-xiaodong yang-xiaodong merged commit 11a3749 into dotnetcore:master Apr 26, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@stubbst stubbst deleted the stubbst:Fix.NeedRetry.GetMessagesFail branch Apr 26, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.