KAFKA-7858: Replace JoinGroup request/response with automated protocol #6419
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
abbccdda
force-pushed
the
join_group_upgrade
branch
7 times, most recently
from
2ee5bdd
to
b68720a
Compare
|
@cmccabe @hachikuji Could you guys give this diff a quick r? This would unblock the progress on KIP-345, thanks! |
abbccdda
force-pushed
the
join_group_upgrade
branch
from
b68720a
to
fbdd1d1
Compare
|
cc @vahidhashemian on this thread since the change is pretty straightforward and requires no background context. |
abbccdda
force-pushed
the
join_group_upgrade
branch
from
fbdd1d1
to
2472a8a
Compare
|
Retest this please |
3 tasks
abbccdda
force-pushed
the
join_group_upgrade
branch
4 times, most recently
from
8126990
to
85ee20f
Compare
|
Retest this please |
abbccdda
force-pushed
the
join_group_upgrade
branch
from
85ee20f
to
dbb400b
Compare
|
@abbccdda: looks good, thanks! When processing older responses, I think generation ID needs to default to -1 to match the previous behavior. This should be changed in |
cmccabe
reviewed
Mar 15, 2019
| @@ -202,7 +203,7 @@ public AbstractCoordinator(LogContext logContext, | |||
| */ | |||
| protected abstract Map<String, ByteBuffer> performAssignment(String leaderId, | |||
| String protocol, | |||
| Map<String, ByteBuffer> allMemberMetadata); | |||
| List<JoinGroupResponseData.JoinGroupResponseMember> allMemberMetadata); | |||
There was a problem hiding this comment.
Should this be a JoinGroupResponseDataSet?
There was a problem hiding this comment.
I don't see this struct anywhere, but I think a list should be fine here?
There was a problem hiding this comment.
Sorry, I meant JoinGroupResponseMemberSet. Anyway, a list is fine too-- I don't feel that strongly about it.
abbccdda
force-pushed
the
join_group_upgrade
branch
from
dbb400b
to
99578dc
Compare
|
Retest this please |
|
@cmccabe @hachikuji mind taking another look? |
|
@cmccabe Another look when you get time? |
|
LGTM. Thanks, @abbccdda . |
|
Thanks! @cmccabe |
3 tasks
ijuma
pushed a commit
that referenced
this pull request
Jul 11, 2019
…p v0 (#7072) The rebalance timeout was added to the JoinGroup protocol in version 1. Prior to 2.3, we handled version 0 JoinGroup requests by setting the rebalance timeout to be equal to the session timeout. We lost this logic when we converted the API to use the generated schema definition (#6419) which uses the default value of -1. The impact of this is that the group rebalance timeout becomes 0, so rebalances finish immediately after we enter the PrepareRebalance state and kick out all old members. This causes consumer groups to enter an endless rebalance loop. This patch restores the old behavior. Reviewers: Ismael Juma <ismael@juma.me.uk>
ijuma
pushed a commit
to confluentinc/kafka
that referenced
this pull request
Jul 11, 2019
…p v0 (apache#7072) The rebalance timeout was added to the JoinGroup protocol in version 1. Prior to 2.3, we handled version 0 JoinGroup requests by setting the rebalance timeout to be equal to the session timeout. We lost this logic when we converted the API to use the generated schema definition (apache#6419) which uses the default value of -1. The impact of this is that the group rebalance timeout becomes 0, so rebalances finish immediately after we enter the PrepareRebalance state and kick out all old members. This causes consumer groups to enter an endless rebalance loop. This patch restores the old behavior. Reviewers: Ismael Juma <ismael@juma.me.uk>
ijuma
pushed a commit
that referenced
this pull request
Jul 11, 2019
…p v0 (#7072) The rebalance timeout was added to the JoinGroup protocol in version 1. Prior to 2.3, we handled version 0 JoinGroup requests by setting the rebalance timeout to be equal to the session timeout. We lost this logic when we converted the API to use the generated schema definition (#6419) which uses the default value of -1. The impact of this is that the group rebalance timeout becomes 0, so rebalances finish immediately after we enter the PrepareRebalance state and kick out all old members. This causes consumer groups to enter an endless rebalance loop. This patch restores the old behavior. Reviewers: Ismael Juma <ismael@juma.me.uk>
xiowu0
pushed a commit
to linkedin/kafka
that referenced
this pull request
Aug 22, 2019
…ession timeout for JoinGroup v0 (apache#7072) TICKET = KAFKA-8653 LI_DESCRIPTION = EXIT_CRITERIA = HASH [b725b3c] ORIGINAL_DESCRIPTION = The rebalance timeout was added to the JoinGroup protocol in version 1. Prior to 2.3, we handled version 0 JoinGroup requests by setting the rebalance timeout to be equal to the session timeout. We lost this logic when we converted the API to use the generated schema definition (apache#6419) which uses the default value of -1. The impact of this is that the group rebalance timeout becomes 0, so rebalances finish immediately after we enter the PrepareRebalance state and kick out all old members. This causes consumer groups to enter an endless rebalance loop. This patch restores the old behavior. Reviewers: Ismael Juma <ismael@juma.me.uk> (cherry picked from commit b725b3c)
singhnama
pushed a commit
to singhnama/kafka
that referenced
this pull request
Jul 20, 2022
*add more validation during KRPC deserialization When deserializing KRPC (which is used for RPCs sent to Kafka, Kafka Metadata records, and some other things), check that we have at least N bytes remaining before allocating an array of size N. Remove DataInputStreamReadable since it was hard to make this class aware of how many bytes were remaining. Instead, when reading an individual record in the Raft layer, simply create a ByteBufferAccessor with a ByteBuffer containing just the bytes we're interested in. Add SimpleArraysMessageTest and ByteBufferAccessorTest. Also add some additional tests in RequestResponseTest. Co-author: Manikumar Reddy <manikumar.reddy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, José Armando García Sancio <jsancio@gmail.com>
omkreddy
pushed a commit
to omkreddy/kafka
that referenced
this pull request
Jul 27, 2022
*add more validation during KRPC deserialization When deserializing KRPC (which is used for RPCs sent to Kafka, Kafka Metadata records, and some other things), check that we have at least N bytes remaining before allocating an array of size N. Remove DataInputStreamReadable since it was hard to make this class aware of how many bytes were remaining. Instead, when reading an individual record in the Raft layer, simply create a ByteBufferAccessor with a ByteBuffer containing just the bytes we're interested in. Add SimpleArraysMessageTest and ByteBufferAccessorTest. Also add some additional tests in RequestResponseTest. Co-author: Manikumar Reddy <manikumar.reddy@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prioritizing this migration because we have blocking feature from KIP-345 part 1: #6177
Upgrade join group protocols could ease the process of adding group instance id towards JoinGroupResponse.
Committer Checklist (excluded from commit message)