KAFKA-5731 Corrected how the sink task worker updates the last committed offsets (0.10.2) #3717
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ted offsets Prior to this change, it was possible for the synchronous consumer commit request to be handled before previously-submitted asynchronous commit requests. If that happened, the out-of-order handlers improperly set the last committed offsets, which then became inconsistent with the offsets the connector task is working with. This change ensures that the last committed offsets are updated only for the most recent commit request, even if the consumer reorders the calls to the callbacks.
Corrects the test case added in the previous commit to properly revoke the existing partition assignments before adding new partition assigments.
# Conflicts: # connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java
# Conflicts: # connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java
…e clarity and usefulness
…tarted Backported the fix for this issue, which was fixed in 0.11.0.0
|
Refer to this link for build results (access rights to CI server needed): |
|
LGTM, thanks for backporting. |
asfgit
pushed a commit
that referenced
this pull request
Aug 23, 2017
…tted offsets Prior to this change, it was possible for the synchronous consumer commit request to be handled before previously-submitted asynchronous commit requests. If that happened, the out-of-order handlers improperly set the last committed offsets, which then became inconsistent with the offsets the connector task is working with. This change ensures that the last committed offsets are updated only for the most recent commit request, even if the consumer reorders the calls to the callbacks. This change also backports the fix for KAFKA-4942, which was minimal that caused the new tests to fail. **This is for the `0.10.2` branch; see #3662 for the equivalent and already-approved PR for `trunk` and #3672 for the equivalent and already-approved PR for the `0.11.0` branch.** Author: Randall Hauch <rhauch@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #3717 from rhauch/kafka-5731-0.10.2
|
@rhauch committed, you'll need to close out the PR |
|
Thanks, @ewencp |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prior to this change, it was possible for the synchronous consumer commit request to be handled before previously-submitted asynchronous commit requests. If that happened, the out-of-order handlers improperly set the last committed offsets, which then became inconsistent with the offsets the connector task is working with.
This change ensures that the last committed offsets are updated only for the most recent commit request, even if the consumer reorders the calls to the callbacks.
This change also backports the fix for KAFKA-4942, which was minimal that caused the new tests to fail.
This is for the
0.10.2branch; see #3662 for the equivalent and already-approved PR fortrunkand #3672 for the equivalent and already-approved PR for the0.11.0branch.