KAFKA-19485: Added check before sending acknowledgements on initial epoch. (#20135)(Cherry-pick) #20670
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…poch. (apache#20135) https://issues.apache.org/jira/browse/KAFKA-19485 **Bug :** There is a bug in `ShareConsumeRequestManager` where we are adding acknowledgements on initial `ShareSession` epoch even after checking for it. Added fix to only include acknowledgements in the request if we have to, PR also adds the check at another point in the code where we could potentially be sending such acknowledgements. One of the cases could be when metadata is refreshed with empty topic IDs after a broker restart. This means leader information would not be available on the node. - Consumer subscribed to a partition whose leader was node-0. - Broker restart happens and node-0 is elected leader again. Broker starts a new `ShareSession`. - Background thread sends a fetch request with **non-zero** epoch. - Broker responds with `SHARE_SESSION_NOT_FOUND`. - Client updates session epoch to 0 once it receives this error. - Client updates metadata but receives empty metadata response. (Leader unavailable) - Application thread processing the previous fetch, completes and sends acks to piggyback on next fetch. - Next fetch will send the piggyback acknowledgements on the fetch for previously subscribed partitions resulting in error from broker ("`Acknowledge data present on initial epoch`"). (Currently we attempt to send even if leader is unavailable). **Fix** : Add a check before sending out acknowledgments if we are on an initial epoch. Added unit test covering the above scenario. Reviewers: Andrew Schofield <aschofield@confluent.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
https://issues.apache.org/jira/browse/KAFKA-19485
Bug :
There is a bug in
ShareConsumeRequestManagerwhere we are addingacknowledgements on initial
ShareSessionepoch even after checking forit.
Added fix to only include acknowledgements in the request if we have to,
PR also adds the check at another point in the code where we could
potentially be sending such acknowledgements. One of the cases could be
when metadata is refreshed with empty topic IDs after a broker restart.
This means leader information would not be available on the node.
starts a new
ShareSession.SHARE_SESSION_NOT_FOUND.unavailable)
acks to piggyback on next fetch.
previously subscribed partitions resulting in error from broker
("
Acknowledge data present on initial epoch"). (Currently we attemptto send even if leader is unavailable).
Fix : Add a check before sending out acknowledgments if we are on
an initial epoch.
Added unit test covering the above scenario.
Reviewers: Andrew Schofield aschofield@confluent.io