[Issue 7554][broker] Split message ranges by ledger ID and store them in individualDeletedMessages #7861
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
massakam
added
type/bug
massakam
force-pushed
the
acked-msg-redelivery
branch
from
f5ae533
to
47c7984
Compare
nkurihar
approved these changes
Aug 26, 2020
massakam
force-pushed
the
acked-msg-redelivery
branch
from
47c7984
to
ea25001
Compare
sijie
approved these changes
Aug 27, 2020
lbenc135
pushed a commit
to lbenc135/pulsar
that referenced
this pull request
Sep 5, 2020
…Messages (apache#7861) Fixes apache#7554 ### Motivation As mentioned in apache#7554, the class of `individualDeletedMessages` is different between Pulsar 2.3.2 (and earlier) and 2.4.0 (and later). This causes some of ranges contained in `individualDeletedMessages` to be lost when the version of Pulsar is upgraded, and a large number of messages that have already been acked can be redelivered to consumers. Also, even if the Pulsar version is 2.4.0 or later, the same phenomenon occurs when the value of `managedLedgerUnackedRangesOpenCacheSetEnabled` is switched from false to true. ### Modifications If the list of individually deleted message ranges loaded from ZK contains ranges that span different ledgers, split the ranges by ledger ID and store them in `individualDeletedMessages`. As a result, information about deleted message ranges is never lost and messages that have already been acked will not be redelivered.
lbenc135
pushed a commit
to lbenc135/pulsar
that referenced
this pull request
Sep 5, 2020
…Messages (apache#7861) Fixes apache#7554 ### Motivation As mentioned in apache#7554, the class of `individualDeletedMessages` is different between Pulsar 2.3.2 (and earlier) and 2.4.0 (and later). This causes some of ranges contained in `individualDeletedMessages` to be lost when the version of Pulsar is upgraded, and a large number of messages that have already been acked can be redelivered to consumers. Also, even if the Pulsar version is 2.4.0 or later, the same phenomenon occurs when the value of `managedLedgerUnackedRangesOpenCacheSetEnabled` is switched from false to true. ### Modifications If the list of individually deleted message ranges loaded from ZK contains ranges that span different ledgers, split the ranges by ledger ID and store them in `individualDeletedMessages`. As a result, information about deleted message ranges is never lost and messages that have already been acked will not be redelivered.
lbenc135
pushed a commit
to lbenc135/pulsar
that referenced
this pull request
Sep 5, 2020
…Messages (apache#7861) Fixes apache#7554 ### Motivation As mentioned in apache#7554, the class of `individualDeletedMessages` is different between Pulsar 2.3.2 (and earlier) and 2.4.0 (and later). This causes some of ranges contained in `individualDeletedMessages` to be lost when the version of Pulsar is upgraded, and a large number of messages that have already been acked can be redelivered to consumers. Also, even if the Pulsar version is 2.4.0 or later, the same phenomenon occurs when the value of `managedLedgerUnackedRangesOpenCacheSetEnabled` is switched from false to true. ### Modifications If the list of individually deleted message ranges loaded from ZK contains ranges that span different ledgers, split the ranges by ledger ID and store them in `individualDeletedMessages`. As a result, information about deleted message ranges is never lost and messages that have already been acked will not be redelivered.
wolfstudy
pushed a commit
that referenced
this pull request
Oct 30, 2020
…Messages (#7861) Fixes #7554 ### Motivation As mentioned in #7554, the class of `individualDeletedMessages` is different between Pulsar 2.3.2 (and earlier) and 2.4.0 (and later). This causes some of ranges contained in `individualDeletedMessages` to be lost when the version of Pulsar is upgraded, and a large number of messages that have already been acked can be redelivered to consumers. Also, even if the Pulsar version is 2.4.0 or later, the same phenomenon occurs when the value of `managedLedgerUnackedRangesOpenCacheSetEnabled` is switched from false to true. ### Modifications If the list of individually deleted message ranges loaded from ZK contains ranges that span different ledgers, split the ranges by ledger ID and store them in `individualDeletedMessages`. As a result, information about deleted message ranges is never lost and messages that have already been acked will not be redelivered. (cherry picked from commit 0b7f034)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #7554
Motivation
As mentioned in #7554, the class of
individualDeletedMessagesis different between Pulsar 2.3.2 (and earlier) and 2.4.0 (and later). This causes some of ranges contained inindividualDeletedMessagesto be lost when the version of Pulsar is upgraded, and a large number of messages that have already been acked can be redelivered to consumers.Also, even if the Pulsar version is 2.4.0 or later, the same phenomenon occurs when the value of
managedLedgerUnackedRangesOpenCacheSetEnabledis switched from false to true.Modifications
If the list of individually deleted message ranges loaded from ZK contains ranges that span different ledgers, split the ranges by ledger ID and store them in
individualDeletedMessages.As a result, information about deleted message ranges is never lost and messages that have already been acked will not be redelivered.