Fixed race condition while triggering message redelivery after an ack-timeout event #5276
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jerrypeng
approved these changes
Sep 25, 2019
|
rerun java8 tests |
|
rerun java8 tests |
1 similar comment
|
rerun java8 tests |
|
run java8 tests |
2 similar comments
|
run java8 tests |
|
run java8 tests |
|
rerun java8 tests |
rdhabalia
reviewed
Sep 30, 2019
| }); | ||
|
|
||
| for (PositionImpl p : pendingPositions) { | ||
| pendingAcks.remove(p.getLedgerId(), p.getEntryId()); |
There was a problem hiding this comment.
pendingPositions will have same number of positions as pendingAcks.. so, why can't we just clear pendingAcks here and anyway, subscription:: redeliverUnacknowledgedMessages is happening later.
There was a problem hiding this comment.
It's to avoid missing any insertion that might be happening into pendingAcks from a different thread. (eg: if the dispatcher thread was actually sending messages at the same time)
|
rerun java8 tests |
rdhabalia
approved these changes
Sep 30, 2019
jiazhai
approved these changes
Oct 7, 2019
|
rerun java8 tests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
There is a race condition when processing a request for redeliveries after an ack-timeout event. The issue is around the update on the
pendingAcksMapinConsumerhandler.When we request to re-deliver all messages, we do:
i. go through
pendingAcksMapand read some of the entriesii. when an entry is read, add the id to
pendingAcksMapand then dispatchi. go through
pendingAcksMapto update stats and then clear the mapIf the reading of the entries to be redelivered is very fast (eg. immediately replayed from cache), the clearing might step in after the dispatching thread has re-added the pending entries.
Therefore, at the next iteration, the broker won't re-deliver these messages which will stay in backlog.
Modifications
Ensure that
pendingAcksMapis correctly updated without a full clear and anyway before any dispatching will take place.