New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KAFKA-16554: Online downgrade triggering and group type conversion #15721

Merged

dajac merged 98 commits into apache:trunk from dongnuo123:online-migration-triggering-downgrade

Apr 25, 2024

Contributor

dongnuo123 commented Apr 15, 2024

Online downgrade from a consumer group to a classic group is triggered when the last consumer that uses the consumer protocol leaves the group. A rebalance is manually triggered after the group conversion.

This patch adds consumer group downgrade validation and conversion.

https://issues.apache.org/jira/browse/KAFKA-16554

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

dongnuo123 and others added 30 commits

February 21, 2024 16:09


          initial push

50c655f


          unit test

92353f5


          unit test

5a1a557


          Merge branch 'trunk' into group-protocol-migration-config

67dc1f3


          rename GroupProtocolMigrationPolicy

980a535


          Merge branch 'trunk' into group-protocol-migration-config

2e91a40


          upgrade conversion

124ba9e


          upgrade conversion

3d7371d


          upgrade conversion

0ace3d8


          draft

9075ea1


          upgrade

b333674


          upgrade

75ff595


          upgrade

1b72303


          unit tests

36aa59c


          unit tests

b9db1f8


          add useLegacyProtocol to ConsumerGroupMember

905cb0c


          addressing comments

52963db


          Merge branch 'apache:trunk' into group-protocol-migration-config

38d7328


          Merge branch 'group-protocol-migration-config' into online-migration-…

f356896

…triggering


          merge config adding branch

eb97afb


          useLegacyProtocol

f1250c6


          Merge branch 'online-migration-triggering' into online-migration-trig…

e0ff833

…gering-downgrade


          blocked by protocol check


          replace useLegacyProtocol with supportedProtocols

c04cff5


          initial downgrade conversion

48ca1a8


          triggering

b532242


          fix unit tests

b33d2d3


          the first unit test

4072e63


          rename

9fb3061


          upgrade conversion

a1edf2f

dongnuo123 added 5 commits

April 14, 2024 16:32


          unit test

9b19011


          serialize version

34397df


          resolve comments

62105d9


          Merge branch 'online-migration-triggering' into online-migration-trig…

…gering-downgrade


          unit tests

aa7f24e

dajac added the KIP-848 label

dongnuo123 added 4 commits

April 16, 2024 10:46


          comments

5a4cf8a


          comments

a03a377


          comments

7bbf833


          Merge branch 'online-migration-triggering' into online-migration-trig…

057e078

…gering-downgrade

dajac reviewed

View reviewed changes

Contributor

dajac left a comment

@dongnuo123 Thanks for the PR. I made a first pass on it and I left some suggestions for consideration.

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java Outdated Show resolved Hide resolved

...p-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java Outdated Show resolved Hide resolved

...p-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java Outdated Show resolved Hide resolved

dongnuo123 added 2 commits

April 17, 2024 09:37


          Merge branch 'trunk' into online-migration-triggering-downgrade

ac04e85


          merge trunk

2a554e9

dajac reviewed

View reviewed changes

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

dongnuo123 added 4 commits

April 17, 2024 13:58


          resolve comments

cf2c2c5


          resolve comments

19dbf7f


          resolve comments

2c7d903


          resolve comments

901d8bf

dajac reviewed

View reviewed changes

Contributor

dajac left a comment

@dongnuo123 Thanks for the update. I left a few more comments.

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

group-coordinator/src/test/java/org/apache/kafka/coordinator/group/Assertions.java Show resolved Hide resolved

...p-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java Outdated Show resolved Hide resolved


          comments

bdb208e

dajac reviewed

View reviewed changes

Contributor

dajac left a comment

@dongnuo123 Thanks for the update. I left another round of comments.

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved

...p-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java Outdated Show resolved Hide resolved

...p-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java Show resolved Hide resolved

...p-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java Outdated Show resolved Hide resolved

...p-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java Show resolved Hide resolved

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Show resolved Hide resolved

dongnuo123 commented

View reviewed changes

group-coordinator/src/test/java/org/apache/kafka/coordinator/group/Assertions.java Outdated Show resolved Hide resolved

dongnuo123 added 2 commits

April 22, 2024 12:09


          addressing comments

509acc9


          completely replace ConsumerGroupFenceMember records

6e2f824

dajac reviewed

View reviewed changes

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java Outdated Show resolved Hide resolved


          move validation to ConsumerGroupFenceMember

2b73350

dajac reviewed

View reviewed changes

Contributor

dajac left a comment

@dongnuo123 Thanks for the update. I left one minor nit and then we can merge it.

group-coordinator/src/test/java/org/apache/kafka/coordinator/group/Assertions.java Outdated Show resolved Hide resolved


          fail the test on a schema exception

9116e43

dajac approved these changes

View reviewed changes

Contributor

dajac left a comment

LGTM, thanks. I will merge it when the CI completes.

dajac merged commit dc1d8fc into apache:trunk

1 check failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment