KAFKA-14591 DeleteRecordsCommand moved to tools #13278

nizhikov · 2023-02-19T15:35:07Z

This PR contains changes to move DeleteRecordsCommand to java code

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

github-actions · 2023-06-10T20:47:58Z

This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch)
If this PR is no longer valid or desired, please feel free to close it. If no activity occurrs in the next 30 days, it will be automatically closed.

fvaleri

Hi @nizhikov, thanks for working on this. Nice job.

It looks like the scala version does not work well when you pass a non existent partition, while yours works as expected.

# trunk
Executing records delete operation
Records delete operation completed:
partition: mytest-0	error: org.apache.kafka.common.errors.TimeoutException: Call(callName=deleteRecords(api=METADATA), deadlineMs=1688824453413, tries=332909, nextAllowedTryMs=1688824453523) timed out at 1688824453423 after 332909 attempt(s)

# this PR
Executing records delete operation
Records delete operation completed:
partition: mytest-0	error: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

Have you considered using the Jackson wrapper proposed here?

Can you please rebase and add a unit test for the various options, like we have for the other tools?

nizhikov · 2023-07-10T09:39:23Z

Hello, @fvaleri

Thanks for the feedback.

Can you please rebase and add a unit test for the various options, like we have for the other tools?

Yes. I merged trunk to the PR and added tests for rewritten command.
Please, take a look into DeleteRecordsCommandTest.

Have you considered using the Jackson wrapper proposed here?

I'm not aware of this work.
It seems right now command works like expected :)

fvaleri

@nizhikov there are a couple of checkstyle errors.

https://ci-builds.apache.org/blue/rest/organizations/jenkins/pipelines/Kafka/pipelines/kafka-pr/branches/PR-13278/runs/3/nodes/10/log/?start=0

Please, take a look into DeleteRecordsCommandTest.

Done. Left some comments.

I'm not aware of this work.
It seems right now command works like expected :)

The JSON parsing logic is shared between multiple tools, including this one, so it seems a nice optimization to have. If we can get #13585 merged soon, then you can also use it here. Let's also see what other think about this.

tools/src/test/java/org/apache/kafka/tools/DeleteRecordsCommandTest.java

nizhikov · 2023-07-10T14:25:42Z

The JSON parsing logic is shared between multiple tools, including this one, so it seems a nice optimization to have.

Sure. We must keep codebase consistent.
Anyway, in the moment my PR seems to be complete, isn't it?

fvaleri · 2023-07-10T15:00:09Z

Yep, LGTM. Just waiting for #13585.

fvaleri · 2023-07-18T09:25:26Z

#13585 has been merged. Can you please rebase and integrate with this PR?

nizhikov · 2023-07-18T11:12:49Z

@fvaleri Done. New tests passed locally.

fvaleri

LGTM. Thanks!

Build failure is due to #14037.

mimaison

Thanks for the PR! I left a few comments.

server-common/src/main/java/org/apache/kafka/server/common/AdminCommandFailedException.java

tools/src/main/java/org/apache/kafka/tools/DeleteRecordsCommand.java

tools/src/test/java/org/apache/kafka/tools/DeleteRecordsCommandTest.java

mimaison · 2023-07-18T14:49:27Z

tools/src/test/java/org/apache/kafka/tools/DeleteRecordsCommandTest.java

+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\"}]}");
+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\", \"partition\": \"\"}]}");
+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\", \"partition\": 0}]}");
+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\", \"offset\":0}]}");


What happens if there are extra fields?

Extra fields will be ignored.

Should we have a test for it?

Success parsing test case extended like this:

@Test public void testParse() throws Exception { Collection<DeleteRecordsCommand.Tuple<TopicPartition, Long>> res = DeleteRecordsCommand.parseOffsetJsonStringWithoutDedup( "{\"partitions\":[" + "{\"topic\":\"t\", \"partition\":0, \"offset\":0}," + "{\"topic\":\"t\", \"partition\":1, \"offset\":1, \"ignored\":\"field\"}," + "{\"topic\":\"t\", \"partition\":0, \"offset\":2}," + "{\"topic\":\"t\", \"partition\":0, \"offset\":0}" + "]}" );

server-common/src/main/java/org/apache/kafka/server/util/CoreUtils.java

server-common/src/test/java/org/apache/kafka/server/util/CoreUtilsTest.java

checkstyle/import-control.xml

nizhikov · 2023-07-18T17:32:17Z

@mimaison Thanks for the review. It seems I addressed all your comments. Please, take a look one more time.

nizhikov · 2023-07-19T04:15:12Z

Tests failures unrelated.

mimaison

Thanks for the updates, I made another pass and left a few more comments

mimaison · 2023-07-19T13:37:42Z

tools/src/main/java/org/apache/kafka/tools/CoreUtils.java

+ * 2. It is the most general possible utility, not just the thing you needed in one particular place
+ * 3. You have tests for it if it is nontrivial in any way
+ */
+public class CoreUtils {


Maybe ToolsUtils would be a better name?

We already have ToolsUtils in server-common, and maybe we should think about moving it to the tools module in a separate PR.

mimaison · 2023-07-19T13:44:02Z

tools/src/main/java/org/apache/kafka/tools/DeleteRecordsCommand.java

+        }
+    }
+
+    public static final class Tuple<V1, V2> {


Do we really need this class? Why can't we use something like Map?

Regular Map implementations can have only one key -> value mapping.
But parseOffsetJsonStringWithoutDedup collects data with possible duplicates.

I can rework command logic and throw on first TopicPartition duplicate, but, it seems out of scope of "just rewrite command in java without changing anything".

Do we have some multi map implementation that can be used here?

We have similar clases like

org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignor.TaskPairs.Pair

org.apache.kafka.clients.consumer.internals.AbstractStickyAssignor.ConsumerGenerationPair

org.apache.kafka.clients.consumer.internals.AbstractStickyAssignor.ConsumerPair

So may be it will be better to keep generic class and reuse it in other places?

I was just asking, if there's a good reason to keep Tuple, we should keep it and not change the tool's semantics.

It seems tool semantic not changed with Map<TopicPartition, List<Long>>.
So current changes conform both semantic requirements and your suggestion.

tools/src/test/java/org/apache/kafka/tools/CoreUtilsTest.java

mimaison · 2023-07-19T13:52:44Z

tools/src/test/java/org/apache/kafka/tools/DeleteRecordsCommandTest.java

+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\"}]}");
+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\", \"partition\": \"\"}]}");
+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\", \"partition\": 0}]}");
+        assertThrowsAdminOperationException("{\"partitions\":[{\"topic\":\"t\", \"offset\":0}]}");


Should we have a test for it?

nizhikov · 2023-07-19T16:03:02Z

@mimaison all your comments addressed except the one with the Tuple. Please, review.

nizhikov · 2023-07-20T07:50:13Z

@mimaison I reworked parseOffsetJsonStringWithoutDedup to return Map<TopicPartition, List<Long>>.
Now, Tuple class eliminated from PR. Please, review.

mimaison

Thanks for all the updates!

nizhikov · 2023-07-21T16:53:08Z

@fvaleri @mimaison Thank you for the review and merge!

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

nizhikov added 5 commits February 17, 2023 16:28

KAFKA-14591 WIP

2695109

KAFKA-14730 AdminOperationException moved to java

a04c84f

Merge branch 'KAFKA-14730' into KAFKA-14591

19ac017

KAFKA-14730 AdminOperationException moved to java

71dd8d6

KAFKA-14591 DeleteRecordsCommand moved to java

f535f4c

nizhikov changed the title ~~KAFKA-14591 DeleteRecordsCommand moved to java~~ KAFKA-14591 DeleteRecordsCommand moved to tools Feb 19, 2023

nizhikov added 2 commits March 8, 2023 13:13

Merge branch 'trunk' into KAFKA-14591

ce12f0c

Merge branch 'trunk' into KAFKA-14591

be834a8

github-actions bot added the stale Stale PRs label Jun 10, 2023

divijvaidya added the tools label Jun 20, 2023

github-actions bot removed the stale Stale PRs label Jun 21, 2023

fvaleri reviewed Jul 8, 2023

View reviewed changes

fvaleri mentioned this pull request Jul 8, 2023

KAFKA-14596: Move TopicCommand to tools #13201

Merged

3 tasks

nizhikov added 3 commits July 9, 2023 13:19

Merge branch 'trunk' into KAFKA-14591

609d9ef

KAFKA-14595 Move new classes to tools

bc4e190

KAFKA-14595 Tests added

c411dbf

fvaleri suggested changes Jul 10, 2023

View reviewed changes

tools/src/test/java/org/apache/kafka/tools/DeleteRecordsCommandTest.java Outdated Show resolved Hide resolved

tools/src/test/java/org/apache/kafka/tools/DeleteRecordsCommandTest.java Outdated Show resolved Hide resolved

fvaleri requested review from mimaison and divijvaidya July 10, 2023 13:21

nizhikov added 2 commits July 10, 2023 17:13

KAFKA-14595 Checkstyle fixes

ad5e0bb

KAFKA-14595 Review fixes

57e308b

nizhikov added 2 commits July 18, 2023 13:53

Merge branch 'trunk' into KAFKA-14591

75098eb

KAFKA-14595 JSon utils used.

b4b21ca

fvaleri approved these changes Jul 18, 2023

View reviewed changes

fvaleri requested review from showuon and removed request for mimaison July 18, 2023 12:03

Merge branch 'trunk' into KAFKA-14591

87278ba

mimaison reviewed Jul 18, 2023

View reviewed changes

KAFKA-14591 Code review fixes

f335c57

mimaison reviewed Jul 19, 2023

View reviewed changes

nizhikov added 2 commits July 19, 2023 18:31

Merge branch 'trunk' into KAFKA-14591

8e3b17b

KAFKA-14591 Code review fixes

e8f5507

KAFKA-14591 Code review fixes

9d185e6

mimaison approved these changes Jul 21, 2023

View reviewed changes

mimaison merged commit 4bba2c8 into apache:trunk Jul 21, 2023
1 check failed

Cerchie pushed a commit to Cerchie/kafka that referenced this pull request Jul 25, 2023

KAFKA-14591: Move DeleteRecordsCommand to tools (apache#13278)

7302324

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

jeqo pushed a commit to aiven/kafka that referenced this pull request Aug 15, 2023

KAFKA-14591: Move DeleteRecordsCommand to tools (apache#13278)

0a01ea7

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-14591 DeleteRecordsCommand moved to tools #13278

KAFKA-14591 DeleteRecordsCommand moved to tools #13278

nizhikov commented Feb 19, 2023

github-actions bot commented Jun 10, 2023

fvaleri left a comment •

edited

nizhikov commented Jul 10, 2023

fvaleri left a comment

nizhikov commented Jul 10, 2023

fvaleri commented Jul 10, 2023

fvaleri commented Jul 18, 2023

nizhikov commented Jul 18, 2023

fvaleri left a comment

mimaison left a comment

mimaison Jul 18, 2023

nizhikov Jul 18, 2023 •

edited

mimaison Jul 19, 2023

nizhikov Jul 19, 2023

nizhikov commented Jul 18, 2023

nizhikov commented Jul 19, 2023

mimaison left a comment

mimaison Jul 19, 2023

fvaleri Jul 19, 2023

nizhikov Jul 19, 2023

mimaison Jul 19, 2023

fvaleri Jul 19, 2023

nizhikov Jul 19, 2023

nizhikov Jul 19, 2023

mimaison Jul 21, 2023

nizhikov Jul 21, 2023

mimaison Jul 19, 2023

nizhikov commented Jul 19, 2023

nizhikov commented Jul 20, 2023

mimaison left a comment

nizhikov commented Jul 21, 2023

KAFKA-14591 DeleteRecordsCommand moved to tools #13278

KAFKA-14591 DeleteRecordsCommand moved to tools #13278

Conversation

nizhikov commented Feb 19, 2023

Committer Checklist (excluded from commit message)

github-actions bot commented Jun 10, 2023

fvaleri left a comment • edited

Choose a reason for hiding this comment

nizhikov commented Jul 10, 2023

fvaleri left a comment

Choose a reason for hiding this comment

nizhikov commented Jul 10, 2023

fvaleri commented Jul 10, 2023

fvaleri commented Jul 18, 2023

nizhikov commented Jul 18, 2023

fvaleri left a comment

Choose a reason for hiding this comment

mimaison left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nizhikov Jul 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nizhikov commented Jul 18, 2023

nizhikov commented Jul 19, 2023

mimaison left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nizhikov commented Jul 19, 2023

nizhikov commented Jul 20, 2023

mimaison left a comment

Choose a reason for hiding this comment

nizhikov commented Jul 21, 2023

fvaleri left a comment •

edited

nizhikov Jul 18, 2023 •

edited