Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added tool allowing changing number of __consumer_offsets topic partitions #7604

Merged
merged 3 commits into from
Jan 3, 2023
Merged

Added tool allowing changing number of __consumer_offsets topic partitions #7604

merged 3 commits into from
Jan 3, 2023

Conversation

mmaslankaprv
Copy link
Member

@mmaslankaprv mmaslankaprv commented Dec 2, 2022 •
edited

Introduced tool that may be helpful when dealing with consumer offsets topic issue. The tool is designed to make changing the __consumer_offsets topic partition count. It can be also used to restore consumer group state.

Signed-off-by: Michal Maslanka michal@redpanda.com

Backports Required

  • none - not a bug fix
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v22.3.x
  • v22.2.x
  • v22.1.x

UX Changes

Release Notes

Features

  • Added a tool allowing changing partition count in __consumer_offsets topic

dotnwat
dotnwat reviewed Dec 2, 2022
View reviewed changes
Copy link
Member

@dotnwat dotnwat left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think it would be better to use an import/export feature from the kafka CLI tooling or even from kcl if it exists? i dunno if it exists or not, but it seems like something that might.

@mmaslankaprv
Copy link
Member Author

do you think it would be better to use an import/export feature from the kafka CLI tooling or even from kcl if it exists? i dunno if it exists or not, but it seems like something that might.

I was asked to provide a script that does not depend on any other tools

@mmaslankaprv mmaslankaprv changed the title tools: introduced simple consumer offsets recovery tool Added tool allowing changing number of __consumer_offsets topic partitions Dec 21, 2022
@mmaslankaprv
tools: introduced simple consumer offsets recovery tool
82d136a 
Introduced tool that may be helpful when dealing with consumer offsets
topic issue. The tool is designed to make changing the
`__consumer_offsets` topic partition count. It can be also used to
restore consumer group state.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv
tests/dockerfile: added consumer group recovery tool to docker image
28c5bae 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv
tests: added test for consumer offsets partition change tool
34e3ff2 
Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv marked this pull request as ready for review December 21, 2022 12:57
@mmaslankaprv mmaslankaprv requested a review from a team as a code owner December 21, 2022 12:57
@mmaslankaprv mmaslankaprv requested review from andrewhsu, dotnwat, daisukebe and a team and removed request for a team December 21, 2022 12:57
@daisukebe
Copy link
Contributor

Pasting the console output for future reference. Just curious if we can suppress the NotCoordinatorForGroupError warning or let it report just once? Given the consumer was stopped the warning itself seems expected though.

$ python3 ./main.py --cfg config.properties -p 16 -r 3 -e
2022-12-22 18:07:57,220 cg-recovery-tool     INFO     starting group recovery tool with options: Namespace(cfg='config.properties', offsets_path='./offsets', verbose=False, skip_query=False, target_partitions=16, execute=True, replication_factor=3)
2022-12-22 18:07:57,220 cg-recovery-tool     INFO     configuration: {'bootstrap_servers': 'localhost'}
2022-12-22 18:07:57,535 cg-recovery-tool     INFO     reading group 'testg' offsets
2022-12-22 18:07:57,537 cg-recovery-tool     INFO     storing offsets in ./offsets
2022-12-22 18:07:57,849 cg-recovery-tool     INFO     removing __consumer_offsets topic
2022-12-22 18:07:57,864 cg-recovery-tool     INFO     recreating __consumer_offsets topic
2022-12-22 18:07:58,301 cg-recovery-tool     INFO     reading offsets from: ./offsets
2022-12-22 18:07:58,406 cg-recovery-tool     INFO     seeking group testg to file offsets/testg.offsets
2022-12-22 18:07:58,510 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:58,714 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:58,818 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:58,920 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,023 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,126 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,229 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,332 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,433 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,536 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,639 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,740 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,845 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:07:59,947 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:08:00,048 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:08:00,155 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:08:00,265 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.
2022-12-22 18:08:00,375 kafka.coordinator    WARNING  Marking the coordinator dead (node coordinator-2) for group testg: [Error 16] NotCoordinatorForGroupError.

@piyushredpanda piyushredpanda added this to the ST.drop milestone Dec 23, 2022
@graphcareful graphcareful self-requested a review December 23, 2022 15:38
graphcareful
graphcareful approved these changes Jan 3, 2023
View reviewed changes
Copy link
Contributor

@graphcareful graphcareful left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tools/consumer_offsets_recovery/main.py
if i == 0:
group = l.strip()
if not dry_run:
consumer = KafkaConsumer(group_id=group, **cfg)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: No need to check if i is 0 within the loop, can just initialize consumer before for loop and perform some error checking against the value of the first line in the file.

tests/rptest/clients/consumer_offsets_recovery.py
from rptest.services.redpanda import RedpandaService


class ConsumerOffsetsRecovery:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super +1

@mmaslankaprv mmaslankaprv merged commit 91e7b10 into redpanda-data:dev Jan 3, 2023
@mmaslankaprv mmaslankaprv mentioned this pull request Jan 4, 2023
Merged
dotnwat added a commit that referenced this pull request Jan 6, 2023
@dotnwat
Merge pull request #8019 from mmaslankaprv/v22.3.x
61b42e7 
[v22.3.x] Backport of #5315 #7603 #7923 #7604 #7814 #8016
This was referenced Mar 10, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@graphcareful graphcareful graphcareful approved these changes

@andrewhsu andrewhsu Awaiting requested review from andrewhsu andrewhsu is a code owner automatically assigned from redpanda-data/devprod

@dotnwat dotnwat Awaiting requested review from dotnwat

@daisukebe daisukebe Awaiting requested review from daisukebe

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
ST.drop
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@mmaslankaprv @daisukebe @dotnwat @graphcareful @piyushredpanda