KAFKA-13452: MM2 shouldn't checkpoint when offset mapping is unavailable #11492
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
viktorsomogyi
force-pushed
the
KAFKA-13452-mm2-checkpoint-issue
branch
2 times, most recently
from
3684020
to
9d0df27
Compare
MM2 checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs - in that case, MM2 simply copies the source offset, which might not be a valid offset in the replica topic at all. This can cause issues when auto offset sync is also turned on. Updated checkpointing logic to not create checkpoints for topic partitions without a valid offset mapping. There is an original feature of copying replica topic offsets back to the upstream cluster. Checkpointing handled this by creating a checkpoint with the original topic name (which was a buggy implementation). These offsets were never meant to be used on the upstream topic, as the offsets probably do not match between the upstream and the downstream topic. Besides that, if there are multiple replication flows, these checkpoints would collide in case sync.group.offsets would be enabled. This functionality is instead kept by transforming the topic name as it went through another hop of replication. This can enable client tools to use the offset-syncs topic of the upstream cluster to map these offsets back.
viktorsomogyi
force-pushed
the
KAFKA-13452-mm2-checkpoint-issue
branch
from
9d0df27
to
35d5b98
Compare
|
@mimaison would you please review this? |
|
Thanks @viktorsomogyi for the PR. If I understand correctly, this an alternative to #11748? |
|
@mimaison yes, it is, if you see something, feel free to incorporate it into yours. Also I didn't want to ping you on this one but a different one :). Sorry. Once I have that up here you can expect me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MM2 checkpointing reads the offset-syncs topic to create offset mappings for committed consumer group offsets. In some corner cases, it is possible that a mapping is not available in offset-syncs - in that case, MM2 simply copies the source offset, which might not be a valid offset in the replica topic at all. This can cause issues when auto offset sync is also turned on.
Updated checkpointing logic to not create checkpoints for topic partitions without a valid offset mapping.
Committer Checklist (excluded from commit message)