KAFKA-6312 Add documentation about kafka-consumer-groups.sh's ability… #4527

tankhiwale · 2018-02-05T15:49:28Z

… to set/change offsets

KIP-122 added the ability for kafka-consumer-groups.sh to reset/change consumer offsets, at a fine grained level.

There is documentation on it in the kafka-consumer-groups.sh usage text.

There is no such documentation on the kafka.apache.org website. We should add some documentation to the website, so that users can read about the functionality without having the tools installed.

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

… to set/change offsets KIP-122 added the ability for kafka-consumer-groups.sh to reset/change consumer offsets, at a fine grained level. There is documentation on it in the kafka-consumer-groups.sh usage text. There is no such documentation on the kafka.apache.org website. We should add some documentation to the website, so that users can read about the functionality without having the tools installed.

tankhiwale · 2018-02-05T15:52:30Z

@omkreddy, @hachikuji I messed up the old pull request (#4496) and created a new one ( rebased with apache/kafka/trunk) for KAFKA-6312
Kindly review. Sincere apologies for confusion. :(

omkreddy · 2018-02-05T16:49:19Z

LGTM

asfgit · 2018-02-05T20:42:28Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-test-coverage/331/

tankhiwale · 2018-02-06T15:11:01Z

@omkreddy, @hachikuji Thank you for the review/inputs. Could you please let me know, in general, what are the next steps involved with merging the PR ? The contributing page says PR will be closed automatically. But i think this will happen when the changes are merged to http://git-wip-us.apache.org/repos/asf/kafka.git (which I don't think I would have rights to)
Lesson I learnt for me was, that I should have created a separate branch for the issue, KAFKA-6312. Anyways, now that there is approval, tankhiwale/trunk is updated, and no multiple commits, do I need to do anything !! Once again, Thank you for your inputs.

hachikuji

Thanks for the patch. Left a few suggestions.

hachikuji · 2018-02-08T01:25:45Z

core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

@@ -900,8 +900,8 @@ object ConsumerGroupCommand extends Logging {
    val CommandConfigDoc = "Property file containing configs to be passed to Admin Client and Consumer."
    val ResetOffsetsDoc = "Reset offsets of consumer group. Supports one consumer group at the time, and instances should be inactive" + nl +
      "Has 3 execution options: (default) to plan which offsets to reset, --execute to execute the reset-offsets process, and --export to export the results to a CSV format." + nl +
-      "Has the following scenarios to choose: --to-datetime, --by-period, --to-earliest, --to-latest, --shift-by, --from-file, --to-current. One scenario must be choose" + nl +
-      "To define the scope use: --all-topics or --topic. . One scope must be choose, unless you use '--from-file' scenario"
+      "Has the following scenarios to choose: --to-datetime, --to-earliest, --to-latest, --shift-by, --from-file, --to-current, --by-duration, --to-offset. One scenario must be choosen" + nl +


typo: "chosen"? or maybe "selected"?
Additionally:

Can we break this line since it is rather long?

Instead of "scenario," how about "reset option"?

hachikuji · 2018-02-08T01:28:11Z

docs/ops.html

@@ -229,6 +229,63 @@ <h4><a id="basic_ops_consumer_group" href="#basic_ops_consumer_group">Managing C
  Deletion of requested consumer groups ('my-group', 'my-other-group') was successful.
  </pre>

+  <p>
+  To reset offsets of a consumer group to latest offset, first make sure the instances are inactive, then:


nit: "to the latest offset"

Also, "instances" is a bit vague. We could say "consumer instances" or maybe just "consumers."

hachikuji · 2018-02-08T01:30:19Z

docs/ops.html

+  topic1                         0          0
+  </pre>
+
+  --reset-offsets supports one consumer group at the time.


Wouldn't it make sense to mention this first before going into the reset to latest example?

hachikuji · 2018-02-08T01:31:31Z

docs/ops.html

+  It has 3 execution options:
+  <ul>
+    <li>
+      (default) to plan which offsets to reset.


I know this is the same language used above, but it's unclear what we mean by "plan." Perhaps a better phrase would be "display"?

hachikuji · 2018-02-08T01:32:50Z

docs/ops.html

+  To reset offsets of a consumer group to latest offset, first make sure the instances are inactive, then:
+
+  <pre class="brush: bash;">
+  &gt; bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group consumergroup1 --topic topic1 --to-latest


Should we be passing --execute here or maybe we should describe above that we are only displaying the offsets that will be reset?

hachikuji · 2018-02-08T01:36:27Z

docs/ops.html

+    </li>
+  </ul>
+
+  --reset-offsets also requires defining following scopes: --all-topics or --topic. One scope must be choosen, unless you use '--from-file' scenario.


I think we should also mention this up-front.

hachikuji · 2018-02-08T01:38:33Z

docs/ops.html

+      --to-latest : Reset offsets to latest offset.
+    </li>
+    <li>
+      --shift-by &lt;Long: number-of-offsets&gt; : Reset offsets shifting current offset by 'n', where 'n' can be positive or negative.


We should probably mention above that out of range offsets will be adjusted to the log start and log end offsets. For example, if the log end offset is 10 and the offset after shifting is 15, then we will actually commit offset 10. Similarly for the other options.

Added a general note at the end of this section.

tankhiwale · 2018-02-08T12:13:46Z

Working on some of @hachikuji 's comments. For rest will get back

hachikuji · 2018-02-24T21:09:40Z

@tankhiwale Any updates? I think it's definitely nice to have some documentation for this tool.

omkreddy · 2018-04-03T07:15:06Z

@tankhiwale Can you address above comments. Let me know, if you need any help.

tankhiwale · 2018-04-04T02:55:15Z

@omkreddy Apologies for leaving this blank. Will try to finish this ASAP. Will get back to you.

tankhiwale · 2018-06-11T07:02:05Z

@hachikuji, @omkreddy this PR is up for review.

omkreddy

Can be include "--by-duration, --to-offset." to kafka-consumer-groups.sh's "--reset-offsets" option
description. This was in your previous commit.

omkreddy · 2018-06-12T11:07:20Z

docs/ops.html

+  </ul>
+
+  <p>
+  For example, to reset offsets of a consumer group to the latest offset:


Can we add this example after explaining the scenarios?

omkreddy · 2018-06-12T11:11:11Z

docs/ops.html

@@ -229,6 +229,69 @@ <h4><a id="basic_ops_consumer_group" href="#basic_ops_consumer_group">Managing C
  Deletion of requested consumer groups ('my-group', 'my-other-group') was successful.
  </pre>

+  <p>
+  To reset offsets of a consumer group, "--reset-offsets" option can be used.
+  --reset-offsets supports one consumer group at the time. It requires defining following scopes: --all-topics or --topic. One scope must be selected, unless you use '--from-file' scenario. Also, first make sure that the consumer instances are inactive.


nit: This options supports .. (no need to repeat --reset-offset)

Can we also include link to KIP docuement.

See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling">KIP-122</a> for more details.

… to set/change offsets Worked on review comments. KIP-122 added the ability for kafka-consumer-groups.sh to reset/change consumer offsets, at a fine grained level. There is documentation on it in the kafka-consumer-groups.sh usage text. There is no such documentation on the kafka.apache.org website. We should add some documentation to the website, so that users can read about the functionality without having the tools installed.

omkreddy

LGTM

cc @hachikuji

tankhiwale · 2018-08-20T06:34:09Z

retest this please

hachikuji

LGTM. Thanks for the patch!

…pache#4527) KIP-122 added the ability for kafka-consumer-groups.sh to reset/change consumer offsets, at a fine grained level. This patch adds documentation for this feature. Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>

hachikuji reviewed Feb 8, 2018

View reviewed changes

ewencp added docs tools labels Feb 28, 2018

Mayank Tankhiwale added 2 commits June 9, 2018 16:21

Merge branch 'apache/kafka' into trunk

9c8d11a

KAFKA-6312 Worked on review comments.

1081646

omkreddy reviewed Jun 12, 2018

View reviewed changes

omkreddy approved these changes Aug 14, 2018

View reviewed changes

hachikuji approved these changes Aug 20, 2018

View reviewed changes

hachikuji merged commit da35cd5 into apache:trunk Aug 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-6312 Add documentation about kafka-consumer-groups.sh's ability… #4527

KAFKA-6312 Add documentation about kafka-consumer-groups.sh's ability… #4527

tankhiwale commented Feb 5, 2018 •

edited

tankhiwale commented Feb 5, 2018

omkreddy commented Feb 5, 2018

asfgit commented Feb 5, 2018

tankhiwale commented Feb 6, 2018 •

edited

hachikuji left a comment

hachikuji Feb 8, 2018

tankhiwale Feb 8, 2018

hachikuji Feb 8, 2018

tankhiwale Feb 8, 2018

hachikuji Feb 8, 2018

tankhiwale Feb 8, 2018

hachikuji Feb 8, 2018

tankhiwale Feb 8, 2018

hachikuji Feb 8, 2018

hachikuji Feb 8, 2018

tankhiwale Feb 8, 2018

hachikuji Feb 8, 2018

tankhiwale Jun 11, 2018

tankhiwale commented Feb 8, 2018

hachikuji commented Feb 24, 2018

omkreddy commented Apr 3, 2018

tankhiwale commented Apr 4, 2018

tankhiwale commented Jun 11, 2018

omkreddy left a comment

omkreddy Jun 12, 2018

tankhiwale Aug 14, 2018

omkreddy Jun 12, 2018

omkreddy Jun 12, 2018 •

edited

tankhiwale Aug 14, 2018

omkreddy left a comment

tankhiwale commented Aug 20, 2018

hachikuji left a comment

KAFKA-6312 Add documentation about kafka-consumer-groups.sh's ability… #4527

KAFKA-6312 Add documentation about kafka-consumer-groups.sh's ability… #4527

Conversation

tankhiwale commented Feb 5, 2018 • edited

Committer Checklist (excluded from commit message)

tankhiwale commented Feb 5, 2018

omkreddy commented Feb 5, 2018

asfgit commented Feb 5, 2018

tankhiwale commented Feb 6, 2018 • edited

hachikuji left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tankhiwale commented Feb 8, 2018

hachikuji commented Feb 24, 2018

omkreddy commented Apr 3, 2018

tankhiwale commented Apr 4, 2018

tankhiwale commented Jun 11, 2018

omkreddy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omkreddy Jun 12, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omkreddy left a comment

Choose a reason for hiding this comment

tankhiwale commented Aug 20, 2018

hachikuji left a comment

Choose a reason for hiding this comment

tankhiwale commented Feb 5, 2018 •

edited

tankhiwale commented Feb 6, 2018 •

edited

omkreddy Jun 12, 2018 •

edited