KAFKA-7196: Remove heartbeat delayed operation for those removed consumers at the end of each rebalance #5556
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…umers at the end of each rebalance
lindong28
changed the title
lindong28
reviewed
Oct 3, 2018
|
|
||
| // the joined member should get heart beat response with no error. Let the new member keep heartbeating for a while | ||
| // to verify that no new rebalance is triggered unexpectedly | ||
| for ( _ <- 1 to 20) { |
There was a problem hiding this comment.
Could you help me understand why we choose 20 iteration and 500 ms per clock advance? What would be the minimum possible iteration to make sure that we catch this bug in the test?
There was a problem hiding this comment.
The minimum possible number of iteration to make sure we catch this bug is 3. Because the session timeout is 1000 ms. That means that the heartbeat of the previous consumer should timeout in 1000 ms and that would trigger another unnecessary rebalance if the heartbeat delayed operation is not removed at the end of the first rebalance.
1000 ms is 2 iterations. So for the code to verify the existence or absence of the error, we just need another extra iteration, which is (2 + 1) = 3.
I chose to make it 20 iteration is to verify the consumer group stays stable in a much longer period of time after the rebalance. Plus that running each iteration takes a negligible amount of time.
There was a problem hiding this comment.
Thanks for the explanation. Sounds good.
|
Thanks for the PR @Lincong. LGTM. Merged to 1.0, 1.1, 2.0 and trunk branches. |
lindong28
pushed a commit
that referenced
this pull request
Oct 4, 2018
…umers at the end of each rebalance During the consumer group rebalance, when the joining group phase finishes, the heartbeat delayed operation of the consumer that fails to rejoin the group should be removed from the purgatory. Otherwise, even though the member ID of the consumer has been removed from the group, its heartbeat delayed operation is still registered in the purgatory and the heartbeat delayed operation is going to timeout and then another unnecessary rebalance is triggered because of it. Author: Lincong Li <lcli@linkedin.com> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5556 from Lincong/remove_heartbeat_delayedOperation (cherry picked from commit 260b07a) Signed-off-by: Dong Lin <lindong28@gmail.com>
lindong28
pushed a commit
that referenced
this pull request
Oct 4, 2018
…umers at the end of each rebalance During the consumer group rebalance, when the joining group phase finishes, the heartbeat delayed operation of the consumer that fails to rejoin the group should be removed from the purgatory. Otherwise, even though the member ID of the consumer has been removed from the group, its heartbeat delayed operation is still registered in the purgatory and the heartbeat delayed operation is going to timeout and then another unnecessary rebalance is triggered because of it. Author: Lincong Li <lcli@linkedin.com> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5556 from Lincong/remove_heartbeat_delayedOperation (cherry picked from commit 260b07a) Signed-off-by: Dong Lin <lindong28@gmail.com>
lindong28
pushed a commit
that referenced
this pull request
Oct 4, 2018
…umers at the end of each rebalance During the consumer group rebalance, when the joining group phase finishes, the heartbeat delayed operation of the consumer that fails to rejoin the group should be removed from the purgatory. Otherwise, even though the member ID of the consumer has been removed from the group, its heartbeat delayed operation is still registered in the purgatory and the heartbeat delayed operation is going to timeout and then another unnecessary rebalance is triggered because of it. Author: Lincong Li <lcli@linkedin.com> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5556 from Lincong/remove_heartbeat_delayedOperation (cherry picked from commit 260b07a) Signed-off-by: Dong Lin <lindong28@gmail.com>
Pengxiaolong
pushed a commit
to Pengxiaolong/kafka
that referenced
this pull request
Jun 14, 2019
…umers at the end of each rebalance During the consumer group rebalance, when the joining group phase finishes, the heartbeat delayed operation of the consumer that fails to rejoin the group should be removed from the purgatory. Otherwise, even though the member ID of the consumer has been removed from the group, its heartbeat delayed operation is still registered in the purgatory and the heartbeat delayed operation is going to timeout and then another unnecessary rebalance is triggered because of it. Author: Lincong Li <lcli@linkedin.com> Reviewers: Dong Lin <lindong28@gmail.com> Closes apache#5556 from Lincong/remove_heartbeat_delayedOperation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
During the consumer group rebalance, when the joining group phase finishes, the heartbeat delayed operation of the consumer that fails to rejoin the group should be removed from the purgatory. Otherwise, even though the member ID of the consumer has been removed from the group, its heartbeat delayed operation is still registered in the purgatory and the heartbeat delayed operation is going to timeout and then another unnecessary rebalance is triggered because of it.
Summary of testing strategy
Extend one of the existing unit test "testRebalanceCompletesBeforeMemberJoins" to verify that the delayed heart beat operation has actually been removed so that it would not trigger another unnecessary group rebalance.
Committer Checklist (excluded from commit message)