KAFKA-14752: Kafka examples improvements #13492

fvaleri · 2023-04-03T06:36:45Z

This is an attempt to improve the Kafka examples module.

There is some heavy refactoring and minor fixes, but the original logic and script interface is kept. Below you can find the list of major changes and fixes:

Make scripts shellcheck compliant
Convert the readme to markdown as in other modules
Remove unnecessary dependency on server-common module
Move all classes from kafka.examples to org.apache.kafka.examples
Add missing and improve existing javadoc
Fix not enough replicas when default minISR>1
Add producer and consumer client id config
Improve log formatting and avoid printing all messages
Fix producer consumer driver hanging on timeout
Remove constructor logic and use try-with-resources
Improve EOS example and processor

Initial set of changes, more will follow with dedicated commits. The intent is to improve but keeping the original examples and interfaces (scripts). - Make scripts shellcheck compliant - Add missing and improve existing javadoc - Fix not enough replicas when default minISR>1 - Add producer and consumer client id config - Removed unnecessary dependency on server-common - Improve log formatting Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

divijvaidya · 2023-04-03T10:49:08Z

May I suggest having different PRs for package change and the actual code changes? It would help view the diff of changes clearly.

fvaleri · 2023-04-03T16:36:30Z

Hi @divijvaidya, I restored the original layout to make review easier.

showuon

@fvaleri , thanks for the PR. It's really important to have a clear example in Kafka. But I think this PR includes many different changes, could we split them out to multiple PRs? I'm thinking we can have:

consumer changes (and utils maybe?)
producer changes
ExactlyOnceMessageProcessor changes
all demo changes
scripts, readme and other remaining changes

WDYT?

showuon · 2023-04-05T09:57:04Z

checkstyle/import-control-core.xml

@@ -71,7 +71,6 @@

  <subpackage name="examples">
    <allow pkg="org.apache.kafka.clients" />
-    <allow pkg="org.apache.kafka.server.util" />


Nice catch!

fvaleri · 2023-04-06T09:19:56Z

@showuon sure, sounds like a plan.

--
I created the additional PRs for better review.

One can still use this one to see the end result and other minor changes (build script, checksyle, etc).

Then, it would also be good to move the class package from kafka.examples to org.apache.kafka.examples as org.apache.kafka seems to be the standard root package prefix in all the other Java modules. This would not be a breaking change because we have wrapper scripts to run the demos.

This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

KAFKA-14752: Kafka examples improvements - consumer changes This is extracted from the original PR for better review. #13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com> Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>

This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri · 2023-05-02T14:05:26Z

This PR was broken into multiple PRs for better review.

This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

…3514) KAFKA-14752: Kafka examples improvements - consumer changes This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com> Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>

commit c757af5 Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri May 12 04:39:12 2023 +0200 KAFKA-14752: Kafka examples improvements - demo changes (apache#13517) KAFKA-14752: Kafka examples improvements - demo changes Reviewers: Luke Chen <showuon@gmail.com> commit 54a4067 Author: Kamal Chandraprakash <kchandraprakash@uber.com> Date: Thu May 11 21:49:21 2023 +0530 KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (apache#13060) Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org> commit bd65db8 Author: Josep Prat <josep.prat@aiven.io> Date: Thu May 11 17:55:26 2023 +0200 MINOR: clean up unused methods in core utils (apache#13706) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org> commit 38adb39 Author: Jeff Kim <kimkb2011@gmail.com> Date: Thu May 11 07:51:41 2023 -0400 MINOR: add test tag for testDeadToDeadIllegalTransition (apache#13694) Reviewers: David Jacot <djacot@confluent.io> commit ee41328 Author: Federico Valeri <fedevaleri@gmail.com> Date: Thu May 11 11:19:32 2023 +0200 KAFKA-14752: Kafka examples improvements - processor changes (apache#13516) Reviewers: Luke Chen <showuon@gmail.com> commit a263627 Author: Mickael Maison <mimaison@users.noreply.github.com> Date: Thu May 11 11:02:45 2023 +0200 MINOR: Remove unused methods in CoreUtils (apache#13170) Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com> commit 920a360 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Thu May 11 04:13:29 2023 +0200 MINOR: fix a small typo in SharedServer.scala (apache#13693) Diabled -> Disabled commit 6d2ad4a Author: A. Sophie Blee-Goldman <sophie@confluent.io> Date: Wed May 10 13:39:15 2023 -0700 HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (apache#13695) The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen() Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io> commit f17fb75 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Wed May 10 16:41:52 2023 +0200 KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (apache#13690) Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 4653507 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Wed May 10 05:09:12 2023 -0700 KAFKA-14514; Add Range Assignor on the Server (KIP-848) (apache#13443) This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers. Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io> commit 625ef17 Author: Luke Chen <showuon@gmail.com> Date: Wed May 10 16:40:20 2023 +0800 MINOR: remove kraft readme link (apache#13691) The config/kraft/README.md is already removed. We should also remove the link. Reviewers: dengziming <dengziming1993@gmail.com> commit 228434d Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue May 9 10:49:27 2023 -0400 KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (apache#13644) This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java. Reviewers: David Jacot <djacot@confluent.io> commit 59ba9db Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 9 17:58:45 2023 +0530 KAFKA-14974: Restore backward compatibility in KafkaBasedLog (apache#13688) `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods. This refactoring changes no functionality other than restoring the older methods. Reviewers: Randall Hauch <rhauch@gmail.com> commit b40a7fc Author: Matthias J. Sax <matthias@confluent.io> Date: Mon May 8 14:24:11 2023 -0700 HOTFIX: fix broken Streams upgrade system test (apache#13654) Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io> commit 7634eee Author: David Jacot <djacot@confluent.io> Date: Mon May 8 20:46:07 2023 +0200 KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (apache#13638) This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state. Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 86daf8c Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Mon May 8 20:09:47 2023 +0530 KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (apache#13594) apache#13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR. Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places. --------- Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 2b98f85 Author: Christo Lolov <lolovc@amazon.com> Date: Mon May 8 15:24:52 2023 +0100 KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (apache#13621) Migrates ChangeLogReader mock in TaskManagerTest to mockito. Reviewer: Bruno Cadonna <cadonna@apache.org> commit 3472389 Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Mon May 8 07:36:36 2023 +0100 KAFKA-14662: Update the ACL list in the doc (apache#13660) Added the missing ACLs to the doc. Reviewers: Luke Chen <showuon@gmail.com> commit a556def Author: Divij Vaidya <diviv@amazon.com> Date: Mon May 8 08:35:14 2023 +0200 MINOR: Print the cause of failure for test (apache#13672) Motivation PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build): org.opentest4j.AssertionFailedError: expected: <false> but was: <true> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31) at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228) at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is. Change 1. Make the test a bit more robust by waiting for server startup. 2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure. Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com> commit 2607e0e Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 08:33:31 2023 +0200 MINOR: Fix producer Callback comment (apache#13669) Fixes the wrong exception name: OffsetMetadataTooLargeException. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com> commit 78090bb Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 04:15:52 2023 +0200 KAFKA-14752: Kafka examples improvements - producer changes (apache#13515) KAFKA-14752: Kafka examples improvements - producer changes Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com> commit 6e7144a Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Sat May 6 02:56:26 2023 +0800 MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (apache#13659) Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro> commit 6bcc497 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 14:05:20 2023 +0200 KAFKA-14766: Improve performance of VarInt encoding and decoding (apache#13312) Motivation Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit. In this PR, we only update varInt32 encoding/decoding. Changes This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation. Results Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com> commit e34f884 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 13:55:17 2023 +0200 KAFKA-14926: Remove metrics on Log Cleaner shutdown (apache#13623) When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io> commit 0822ce0 Author: David Arthur <mumrah@gmail.com> Date: Fri May 5 04:35:26 2023 -0400 KAFKA-14840: Support for snapshots during ZK migration (apache#13461) This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active controller loaded a snapshot, it would get out of sync with the ZK state. In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass and without using too much memory. Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com> commit 97c36f3 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 12:20:33 2023 -0700 HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging apache#13540 commit 63f9f23 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 11:18:03 2023 -0700 MINOR: improve QuorumController logging apache#13540 When creating the QuorumController, log whether ZK migration is enabled. When applying a feature level record which sets the metadata version, log the metadata version enum rather than the numeric feature level. Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the end of replay. When TRACE is enabled, log every record that is replayed in QuorumController. Since some records may contain sensitive information, create RecordRedactor to assist in logging only what is safe to put in the log4j file. Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged at INFO, and additions are logged at TRACE. Remove SnapshotReason.java, SnapshotReasonTest.java, and QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to org.apache.kafka.image.publisher.SnapshotGenerator. Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit ffd814d Author: Justine Olshan <jolshan@confluent.io> Date: Thu May 4 09:55:45 2023 -0700 KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (apache#13607) Also modifies verification to only add a partition to verify if it is transactional. When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different. Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit ea81e99 Author: Philip Nee <pnee@confluent.io> Date: Thu May 4 09:20:01 2023 -0700 KAFKA-13668: Retry upon missing initProducerId due to authorization error (apache#12149) Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer. This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin. Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest. Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dc7819d Author: Christo Lolov <lolovc@amazon.com> Date: Thu May 4 11:00:33 2023 +0100 KAFKA-14594: Move LogDirsCommand to tools module (apache#13122) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d46c3f2 Author: David Mao <47232755+splett2@users.noreply.github.com> Date: Wed May 3 17:09:43 2023 -0700 MINOR: Reduce number of threads created for integration test brokers (apache#13655) The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker. Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io> commit c08120f Author: Jason Gustafson <jason@confluent.io> Date: Wed May 3 15:25:32 2023 -0700 MINOR: Allow tagged fields with version subset of flexible version range (apache#13551) The generated message types are missing a range check for the case when the tagged version range is a subset of the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the number of tags serialized. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b620c03 Author: Luke Chen <showuon@gmail.com> Date: Thu May 4 01:08:25 2023 +0800 KAFKA-14946: fix NPE when merging the deltatable (apache#13653) Fix NPE while merging the deltatable. Because it's possible that hashTier is not null but deltatable is null (ex: removing data), we should have null check while merging for deltatable like other places did. Also added tests that will fail without this change. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 4a0b6eb Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 2 23:16:46 2023 +0530 KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (apache#13657) Reviewers: Chris Egerton <chrise@aiven.io> commit 16fc8e1 Author: David Jacot <djacot@confluent.io> Date: Tue May 2 18:04:50 2023 +0200 KAFKA-14462; [10/N] Add TargetAssignmentBuilder (apache#13637) This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group. Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit f44ee4f Author: Christo Lolov <lolovc@amazon.com> Date: Tue May 2 16:39:31 2023 +0100 MINOR: Remove unnecessary code in client/connect (apache#13259) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 33012b5 Author: Federico Valeri <fedevaleri@gmail.com> Date: Tue May 2 14:28:42 2023 +0200 KAFKA-14752: Kafka examples improvements - consumer changes (apache#13514) KAFKA-14752: Kafka examples improvements - consumer changes This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com> Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com> commit 141c76a Author: Bruno Cadonna <cadonna@apache.org> Date: Tue May 2 14:00:34 2023 +0200 KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (apache#13529) 1. Migrates topology builder mock in TaskManagerTest to mockito. 2. Replaces the unit test to verify if subscribed partitions are added to topology metadata. 3. Modifies signatures of methods for adding subscribed partitions to topology metadata to use sets instead of lists. This makes the intent of the methods clearer and makes the tests more portable. Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org> commit 21af191 Author: Luke Chen <showuon@gmail.com> Date: Tue May 2 09:54:12 2023 +0800 MINOR: Add reason to exceptions in QuorumController (apache#13648) Saw this error message in log: ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController) The null reason is not helpful with only RejectedExecutionException. Adding the reason to it. Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit 4773961 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon May 1 21:51:09 2023 +0530 MINOR: Fix Javadoc for configureAdminResources in Connect's RestServer (apache#13635) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io> commit e299423 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Mon May 1 09:56:04 2023 -0400 KAFKA-14859: SCRAM ZK to KRaft migration with dual write (apache#13628) Handle migrating SCRAM records in ZK when migrating from ZK to KRaft. This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes. Reviewers: David Arthur <mumrah@gmail.com> commit 64ebbc5 Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 08:46:40 2023 -0700 MINOR: Fixing typos in the ConsumerCoordinator (apache#13618) Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io> commit e55fbce Author: Anton Agestam <anton.agestam@aiven.io> Date: Fri Apr 28 11:54:28 2023 +0100 MINOR: Fix incorrect description of SessionLifetimeMs (apache#13649) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit c6ad151 Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 02:08:32 2023 -0700 KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (apache#13550) This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS. This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method. In short, it should no long check and validate only the highest generation. Instead, we consider 3 cases: 1. Member will continue to hold on to its partition if there are no other owners 2. If there are 1+ owners to the same partition. One with the highest generation will win. 3. If two members of the same generation hold on to the same partition. We will log an error but remove both from the assignment. (Same with the current logic) Here are some important notes that lead to the patch: - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown. - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error. This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive To summarize the current behavior of different errors: `REBALANCE_IN_PROGRESS` - heartbeat: requestRejoin if member state is stable - joinGroup: rejoin immediately - syncGroup: rejoin immediately - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress. `UNKNOWN_MEMBER_ID` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately - syncGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately `ILLEGAL_GENERATION` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing. then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately Reviewers: David Jacot <djacot@confluent.io> commit 10b3e66 Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri Apr 28 10:32:11 2023 +0200 KAFKA-14584: Deprecate StateChangeLogMerger tool (apache#13171) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d796480 Author: Luke Chen <showuon@gmail.com> Date: Fri Apr 28 14:35:12 2023 +0800 KAFKA-14909: check zkMigrationReady tag before migration (apache#13631) 1. add ZkMigrationReady in apiVersionsResponse 2. check all nodes if ZkMigrationReady are ready before moving to next migration state Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com> commit c708f7b Author: Colin Patrick McCabe <cmccabe@apache.org> Date: Thu Apr 27 19:15:26 2023 -0700 MINOR: remove spurious call to fatalFaultHandler (apache#13651) Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805. We should only invoke the fatal fault handller if we are unable to generate the activation records. If we are unable to write the activation records, a controller failover should be sufficient to remedy the situation. Co-authored-by: Luke Chen showuon@gmail.com Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com> commit 056657d Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 17:04:56 2023 -0400 MINOR add license to reviewers.py commit a08f31e Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 14:32:19 2023 -0400 MINOR: Adding reviewers.py (apache#11096) This script can be used to help build the "Reviewers: " string we include in commit messages. Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit 7049333 Author: Colin P. McCabe <cmccabe@apache.org> Date: Wed Apr 26 16:10:46 2023 -0700 KAFKA-14943: Fix ClientQuotaControlManager validation Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of internal ZK representation details that treated them both as "client configs.") Add unit tests for ClientQuotaControlManager.isValidIpEntity and ClientQuotaControlManager.configKeysForEntityType. This change doesn't affect metadata record application, only input validation. If there are bad client quotas that are set currently, this change will not alter the current behavior (of throwing an exception and ignoring the bad quota). commit 8bde4e7 Author: David Jacot <djacot@confluent.io> Date: Thu Apr 27 14:05:41 2023 +0200 KAFKA-14462; [9/N] Add RecordHelpers (apache#13544) This patch adds RecordHelpers. Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dd6690a Author: LinShunKang <linshunkang.chn@gmail.com> Date: Thu Apr 27 10:44:08 2023 +0800 KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (apache#12545) This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035 Direct use ByteBuffer instead of byte[] to deserialize. Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro> commit c1b5c75 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 26 10:20:30 2023 -0400 KAFKA-14805 KRaft controller supports pre-migration mode (apache#13407) This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode, the controller will only allow certain write operations. The purpose of this is to disallow metadata changes when the controller is waiting for the ZK migration records to be committed. The following ControllerWriteEvent operations are permitted in pre-migration mode * completeActivation * maybeFenceReplicas * writeNoOpRecord * processBrokerHeartbeat * registerBroker (only for migrating ZK brokers) * unregisterBroker Raft events and other controller events do not follow the same code path as ControllerWriteEvent, so they are not affected by this new behavior. This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean MIGRATION rather than PRE_MIGRATION. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 007c0d3 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 26 17:53:56 2023 +0530 KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (apache#13634) Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit baf127a Author: Greg Harris <greg.harris@aiven.io> Date: Wed Apr 26 00:30:13 2023 -0700 KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (apache#13429) Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io> commit ced1f62 Author: Victoria Xia <victoria.xia@confluent.io> Date: Tue Apr 25 22:39:23 2023 -0400 KAFKA-14834: [13/N] Docs updates for versioned store semantics (apache#13622) Reviewers: Matthias J. Sax <matthias@confluent.io> commit a7d0b3f Author: Said Boudjelda <bmscomp@gmail.com> Date: Tue Apr 25 23:31:04 2023 +0200 MINOR: Upgrade gradle to 8.1.1 (apache#13625) Also upgrade gradle plugins: - `org.owasp.dependencycheck` gradle plugin to version `8.2.1` - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1` Gradle release notes: * https://docs.gradle.org/8.1.1/release-notes.html Reviewers: Ismael Juma <ismael@juma.me.uk> commit 9a36da1 Author: David Jacot <djacot@confluent.io> Date: Tue Apr 25 18:50:51 2023 +0200 KAFKA-14462; [8/N] Add ConsumerGroupMember (apache#13538) This patch adds ConsumerGroupMember. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io> commit 4780dc7 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue Apr 25 21:20:35 2023 +0530 KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (apache#13636) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit ea540fa Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Tue Apr 25 13:28:37 2023 +0100 KAFKA-14592: Move FeatureCommand to tools (apache#13459) KAFKA-14592: Move FeatureCommand to tools Reviewers: Luke Chen <showuon@gmail.com> commit d83a734 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 25 12:02:24 2023 +0200 MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (apache#13626) The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call. Reviewers: maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com> commit 2557a4b Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 15:29:57 2023 -0700 KAFKA-12446: update change encoding to use varint (apache#13533) KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io> commit ab8f285 Author: Victoria Xia <victoria.xia@confluent.io> Date: Mon Apr 24 17:06:26 2023 -0400 KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (apache#13615) Reviewers: Matthias J. Sax <matthias@confluent.io> commit 6dcdb01 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 12:40:25 2023 -0700 KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (apache#13592) Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks. This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io> commit 2271e74 Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Mon Apr 24 17:21:19 2023 +0800 MINOR: fix zookeeper_migration_test.py (apache#13620) Reviewers: Mickael Maison <mimaison@users.noreply.github.com> commit c324123 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon Apr 24 14:06:20 2023 +0530 KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (apache#13587) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 7061475 Author: Greg Harris <greg.harris@aiven.io> Date: Fri Apr 21 12:55:41 2023 -0700 KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (apache#13575) Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest` Reviewers: Josep Prat <jlprat@apache.org> commit ecdef88 Author: Matthias J. Sax <matthias@confluent.io> Date: Fri Apr 21 12:48:05 2023 -0700 MINOR: updated KS release notes for 3.5 (apache#13577) Reviewers: Walker Carlson <wcarlson@confluent.io> commit dd63d88 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Fri Apr 21 15:02:06 2023 +0200 MINOR: fix noticed typo in raft and metadata projects (apache#13612) Reviewers: Josep Prat <jlprat@apache.org> commit c39bf71 Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:22:16 2023 +0200 KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (apache#13537) This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 2d0b816 Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:19:04 2023 +0200 MINOR: Move `ControllerPurgatory` to `server-common` (apache#13555) This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit df13775 Author: Purshotam Chauhan <pchauhan@confluent.io> Date: Fri Apr 21 14:08:23 2023 +0530 KAFKA-14828: Remove R/W locks using persistent data structures (apache#13437) Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only. In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation. Reviewers: Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Divij Vaidya <diviv@amazon.com> commit 2ee770a Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu Apr 20 10:21:26 2023 -0700 Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (apache#13572)" This reverts commit d04c3e5. commit ef09a2e Author: Justine Olshan <jolshan@confluent.io> Date: Thu Apr 20 09:30:11 2023 -0700 KAFKA-14904: Pending state blocked verification of transactions (apache#13579) KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction. There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes. When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions. The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely. This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed. Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit 7f175fe Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Thu Apr 20 13:50:29 2023 +0530 KAFKA-14586: Adding redirection for StreamsResetter (apache#13614) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com> commit e14dd80 Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com> Date: Thu Apr 20 05:29:27 2023 +0200 KAFKA-14821 Implement the listOffsets API with AdminApiDriver (apache#13432) We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic: 1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry. 2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of the client. 3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`. We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them. Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io> commit f5de4da Author: Ron Dagostino <rndgstn@gmail.com> Date: Wed Apr 19 19:45:26 2023 -0400 KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change events. However, it does not distinguish between issues related to feature changes actually failing and other exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues. We shutdown the broker only for FeatureCacheUpdateException. Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 11c8bf4 Author: Victoria Xia <victoria.xia@confluent.io> Date: Wed Apr 19 19:34:36 2023 -0400 KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (apache#13609) This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter. Part of KIP-914. Reviewer: Matthias J. Sax <matthias@confluent.io> commit 809966a Author: Matthew de Detrich <matthew.dedetrich@aiven.io> Date: Wed Apr 19 20:54:07 2023 +0200 KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (apache#11478) Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address. Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org> commit 750cfd8 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 19 14:19:13 2023 -0400 KAFKA-14918 Only send controller RPCs to migrating ZK brokers (apache#13606) This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b10716e Author: hudeqi <1217150961@qq.com> Date: Thu Apr 20 00:49:08 2023 +0800 KAFKA-14868: Remove all ReplicaManager metrics when it is closed (apache#13471) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com> commit d04c3e5 Author: Keith Wall <kwall@redhat.com> Date: Wed Apr 19 03:58:29 2023 +0100 KAFKA-14908: Set setReuseAddress on the kafka server socket (apache#13572) Changes SocketServer to set the setReuseAddress(true) socket option. This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test. Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com> commit f905a5a Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 19 00:46:03 2023 +0530 MINOR: Fixing gradle build during compileScala and compileTestScala (apache#13588) Reviewers: Chia-Ping Tsai <chia7712@gmail.com> commit 3388adf Author: Matthias J. Sax <matthias@confluent.io> Date: Tue Apr 18 11:32:27 2023 -0700 MINOR: rename internal FK-join processor classes (apache#13589) Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io> commit abca865 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Tue Apr 18 12:41:38 2023 -0400 KAFKA-14881: Rework UserScramCredentialRecord (apache#13513) Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This is necessary to support migration from ZK, since those are the fields we stored in ZK. Update latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version. Moved ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more appropriate. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 61530d6 Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 09:37:04 2023 -0400 KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (apache#13526) This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields. Reviewers: David Jacot <djacot@confluent.io> commit b36a170 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 18 13:36:56 2023 +0200 MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (apache#13593) Reviewers: Luke Chen <showuon@gmail.com> commit 5e9d4de Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 04:41:54 2023 -0400 KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (apache#13511) This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io> commit 454b721 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Tue Apr 18 09:40:14 2023 +0200 KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (apache#13557) KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow. To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread. commit e27926f Author: Ron Dagostino <rndgstn@gmail.com> Date: Mon Apr 17 17:52:28 2023 -0400 KAFKA-14735: Improve KRaft metadata image change performance at high … (apache#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io> commit 7159f6c Author: Alyssa Huang <ahuang@confluent.io> Date: Mon Apr 17 11:03:45 2023 -0700 MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it should have been. Add a regression test. Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit 061dac797bae56d91659fbc8d7900117eb10a866 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:43:12 2023 -0700 Moved SubscribedTopicMetadata to the consumer package commit 6650ea5cd707f03713fc2c69ba76fafd56d886b8 Merge: 22f2df02de faafed25a1 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:33:17 2023 -0700 Merge remote-tracking branch 'origin/rreddy-22/Rack-Assignor-Interface-Changes' into rreddy-22/Rack-Assignor-Interface-Changes commit 22f2df02deabe1a6b5b8b8cceb432486b9ef4470 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:25:11 2023 -0700 minor changes commit faafed25a10d7ff79fb8a24ab62aa222c0e6516a Merge: 4511683d4e 1bf73d89d0 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Fri Jul 21 12:50:59 2023 -0700 Merge branch 'trunk' into rreddy-22/Rack-Assignor-Interface-Changes commit 4511683d4ef489cb979769ff6775dfe0250cdeeb Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 21 03:09:27 2023 -0700 Changes based on PR comments commit 579bb948fcc78afd163410a5bc43832bc87aaf71 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Jul 19 14:45:18 2023 -0700 New interfaces and classes added to facilitate passing rack information to the assignor, modified tests to incorporate changes commit f68471a99f2e0260b36047a7596b0f9de48794ad Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Jul 13 11:55:20 2023 -0700 Small edits commit c50f3bdbf79f1ca127355d2a9002b1fdbf0baa2f Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 11 16:30:32 2023 -0700 Removed extra lines commit 478488e0943aa879b27eec5ae5eb79a23dd9cab3 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 11 16:21:54 2023 -0700 Added rack interface changes, added a new TopicAndClusterMetadata class to handle topic and cluster images and added an AbstractPartitionAssignor commit ce540f943b8dae0786a87e51cafacb66efd1e64d Merge: 9815d6db04 fd5b300b57 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:45:27 2023 -0600 Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized commit 9815d6db04c264456b7ead7fc5f735513a868c7d Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:23:00 2023 -0600 Abstract Partition Assignor commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Jun 29 16:36:53 2023 -0700 Refactored code to reduce format conversion time commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 31 16:21:01 2023 -0700 test changes and code changes commit e1a032579129a2cd3aae1e93f46570bd57d3343c Author: Ritika Reddy <rreddy@confluent.io> Date: Tue May 16 09:28:57 2023 -0700 small changes commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu May 11 20:36:59 2023 -0700 Squashed commit of the following: commit c757af5f7c630d532bfee5f6dc45aec603ad8a29 Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri May 12 04:39:12 2023 +0200 KAFKA-14752: Kafka examples improvements - demo changes (#13517) KAFKA-14752: Kafka examples improvements - demo changes Reviewers: Luke Chen <showuon@gmail.com> commit 54a4067f81e1434d956ef797274f7b437fe49ea1 Author: Kamal Chandraprakash <kchandraprakash@uber.com> Date: Thu May 11 21:49:21 2023 +0530 KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060) Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org> commit bd65db82b4bad623b0bb31398979e466978148da Author: Josep Prat <josep.prat@aiven.io> Date: Thu May 11 17:55:26 2023 +0200 MINOR: clean up unused methods in core utils (#13706) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org> commit 38adb3956979d7025d148612975cc0b82200b2e1 Author: Jeff Kim <kimkb2011@gmail.com> Date: Thu May 11 07:51:41 2023 -0400 MINOR: add test tag for testDeadToDeadIllegalTransition (#13694) Reviewers: David Jacot <djacot@confluent.io> commit ee4132863553fb4fd2df715b2fbd77f349f978b8 Author: Federico Valeri <fedevaleri@gmail.com> Date: Thu May 11 11:19:32 2023 +0200 KAFKA-14752: Kafka examples improvements - processor changes (#13516) Reviewers: Luke Chen <showuon@gmail.com> commit a263627adb75f1ca5c87f1482cc70b994ba49d63 Author: Mickael Maison <mimaison@users.noreply.github.com> Date: Thu May 11 11:02:45 2023 +0200 MINOR: Remove unused methods in CoreUtils (#13170) Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com> commit 920a3601ffa7266edf12f3559cfe97c8a5929d03 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Thu May 11 04:13:29 2023 +0200 MINOR: fix a small typo in SharedServer.scala (#13693) Diabled -> Disabled commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc Author: A. Sophie Blee-Goldman <sophie@confluent.io> Date: Wed May 10 13:39:15 2023 -0700 HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695) The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen() Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io> commit f17fb75b2de32512f14cb94a7d1bfb0f37485780 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Wed May 10 16:41:52 2023 +0200 KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690) Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 4653507926a42dccda5c086fcae6278afcfc53ca Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Wed May 10 05:09:12 2023 -0700 KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443) This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers. Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io> commit 625ef176ee5f167786003c2d88498632b0b7014b Author: Luke Chen <showuon@gmail.com> Date: Wed May 10 16:40:20 2023 +0800 MINOR: remove kraft readme link (#13691) The config/kraft/README.md is already removed. We should also remove the link. Reviewers: dengziming <dengziming1993@gmail.com> commit 228434d23583189cdcaa7f4a90ebb178ccc17c73 Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue May 9 10:49:27 2023 -0400 KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644) This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java. Reviewers: David Jacot <djacot@confluent.io> commit 59ba9dbbc927ddc8660d0d98d9422909fd306758 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 9 17:58:45 2023 +0530 KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688) `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods. This refactoring changes no functionality other than restoring the older methods. Reviewers: Randall Hauch <rhauch@gmail.com> commit b40a7fc037bb1543c3355fad9c71570f770f5177 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon May 8 14:24:11 2023 -0700 HOTFIX: fix broken Streams upgrade system test (#13654) Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io> commit 7634eee2627da39937e3112ffc58bd7cfedc98f2 Author: David Jacot <djacot@confluent.io> Date: Mon May 8 20:46:07 2023 +0200 KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638) This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state. Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 86daf8ce6573eb79d6e78381dbab738055f914c4 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Mon May 8 20:09:47 2023 +0530 KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594) #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR. Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places. --------- Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6 Author: Christo Lolov <lolovc@amazon.com> Date: Mon May 8 15:24:52 2023 +0100 KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621) Migrates ChangeLogReader mock in TaskManagerTest to mockito. Reviewer: Bruno Cadonna <cadonna@apache.org> commit 347238948b86882a47faee4a2916d1b01333d95f Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Mon May 8 07:36:36 2023 +0100 KAFKA-14662: Update the ACL list in the doc (#13660) Added the missing ACLs to the doc. Reviewers: Luke Chen <showuon@gmail.com> commit a556def5724efb1dc96bd2d389411a8d2f802f53 Author: Divij Vaidya <diviv@amazon.com> Date: Mon May 8 08:35:14 2023 +0200 MINOR: Print the cause of failure for test (#13672) Motivation PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build): org.opentest4j.AssertionFailedError: expected: <false> but was: <true> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31) at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228) at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is. Change 1. Make the test a bit more robust by waiting for server startup. 2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure. Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com> commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 08:33:31 2023 +0200 MINOR: Fix producer Callback comment (#13669) Fixes the wrong exception name: OffsetMetadataTooLargeException. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com> commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399 Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 04:15:52 2023 +0200 KAFKA-14752: Kafka examples improvements - producer changes (#13515) KAFKA-14752: Kafka examples improvements - producer changes Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com> commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714 Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Sat May 6 02:56:26 2023 +0800 MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659) Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro> commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 14:05:20 2023 +0200 KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312) Motivation Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit. In this PR, we only update varInt32 encoding/decoding. Changes This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation. Results Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com> commit e34f88403159cc8381da23dafdf7e3d7403114a2 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 13:55:17 2023 +0200 KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623) When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io> commit 0822ce0ed1a106a510930bc9ac53a266f54684d7 Author: David Arthur <mumrah@gmail.com> Date: Fri May 5 04:35:26 2023 -0400 KAFKA-14840: Support for snapshots during ZK migration (#13461) This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active controller loaded a snapshot, it would get out of sync with the ZK state. In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass and without using too much memory. Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com> commit 97c36f3f3142580325daa1a6aadb662893390561 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 12:20:33 2023 -0700 HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540 commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 11:18:03 2023 -0700 MINOR: improve QuorumController logging #13540 When creating the QuorumController, log whether ZK migration is enabled. When applying a feature level record which sets the metadata version, log the metadata version enum rather than the numeric feature level. Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the end of replay. When TRACE is enabled, log every record that is replayed in QuorumController. Since some records may contain sensitive information, create RecordRedactor to assist in logging only what is safe to put in the log4j file. Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged at INFO, and additions are logged at TRACE. Remove SnapshotReason.java, SnapshotReasonTest.java, and QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to org.apache.kafka.image.publisher.SnapshotGenerator. Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff Author: Justine Olshan <jolshan@confluent.io> Date: Thu May 4 09:55:45 2023 -0700 KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607) Also modifies verification to only add a partition to verify if it is transactional. When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different. Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit ea81e99e5980c807414651034a8c60426a158ca4 Author: Philip Nee <pnee@confluent.io> Date: Thu May 4 09:20:01 2023 -0700 KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149) Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer. This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin. Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest. Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dc7819d7f1fe6b0160cd95246420ab10c335410b Author: Christo Lolov <lolovc@amazon.com> Date: Thu May 4 11:00:33 2023 +0100 KAFKA-14594: Move LogDirsCommand to tools module (#13122) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d46c3f259cce25c43f20fba3943d5cb34ed909ea Author: David Mao <47232755+splett2@users.noreply.github.com> Date: Wed May 3 17:09:43 2023 -0700 MINOR: Reduce number of threads created for integration test brokers (#13655) The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker. Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io> commit c08120f83f7318f15dcf14d525876d18caf6afd0 Author: Jason Gustafson <jason@confluent.io> Date: Wed May 3 15:25:32 2023 -0700 MINOR: Allow tagged fields with version subset of flexible version range (#13551) The generated message types are missing a range check for the case when the tagged version range is a subset of the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the number of tags serialized. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b620c03ccf48d6d92b219cba35bb1e5e248d2547 Author: Luke Chen <showuon@gmail.com> Date: Thu May 4 01:08:25 2023 +0800 KAFKA-14946: fix NPE when merging the deltatable (#13653) Fix NPE while merging the deltatable. Because it's possible that hashTier is not null but deltatable is null (ex: removing data), we should have null check while merging for deltatable like other places did. Also added tests that will fail without this change. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 4a0b6ebf60ed7614f042443460b490971e8662a4 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 2 23:16:46 2023 +0530 KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657) Reviewers: Chris Egerton <chrise@aiven.io> commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e Author: David Jacot <djacot@confluent.io> Date: Tue May 2 18:04:50 2023 +0200 KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637) This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group. Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949 Author: Christo Lolov <lolovc@amazon.com> Date: Tue May 2 16:39:31 2023 +0100 MINOR: Remove unnecessary code in client/connect (#13259) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3 Author: Federico Valeri <fedevaleri@gmail.com> Date: Tue May 2 14:28:42 2023 +0200 KAFKA-14752: Kafka examples improvements - consumer changes (#13514) KAFKA-14752: Kafka examples improvements - consumer changes This is extracted from the original PR for better review. https://github.com/apache/kafka/pull/13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com> Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com> commit 141c76a2c904705f2cd484e96767fcb217c5db25 Author: Bruno Cadonna <cadonna@apache.org> Date: Tue May 2 14:00:34 2023 +0200 KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529) 1. Migrates topology builder mock in TaskManagerTest to mockito. 2. Replaces the unit test to verify if subscribed partitions are added to topology metadata. 3. Modifies signatures of methods for adding subscribed partitions to topology metadata to use sets instead of lists. This makes the intent of the methods clearer and makes the tests more portable. Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org> commit 21af1918eafa30812a955c3c0295b9e968841cd3 Author: Luke Chen <showuon@gmail.com> Date: Tue May 2 09:54:12 2023 +0800 MINOR: Add reason to exceptions in QuorumController (#13648) Saw this error message in log: ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController) The null reason is not helpful with only RejectedExecutionException. Adding the reason to it. Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit 4773961a44d1f0d1e11d662c7e0fc955027bced2 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon May 1 21:51:09 2023 +0530 MINOR: Fix Javadoc for configureAdminResources in Connect's RestServer (#13635) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io> commit e29942347acc70aa85d47e84e2021f9c24cd7c80 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Mon May 1 09:56:04 2023 -0400 KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628) Handle migrating SCRAM records in ZK when migrating from ZK to KRaft. This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes. Reviewers: David Arthur <mumrah@gmail.com> commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 08:46:40 2023 -0700 MINOR: Fixing typos in the ConsumerCoordinator (#13618) Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io> commit e55fbceb6667cc1455f7fb2e96421c85741fa7df Author: Anton Agestam <anton.agestam@aiven.io> Date: Fri Apr 28 11:54:28 2023 +0100 MINOR: Fix incorrect description of SessionLifetimeMs (#13649) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984 Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 02:08:32 2023 -0700 KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550) This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS. This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method. In short, it should no long check and validate only the highest generation. Instead, we consider 3 cases: 1. Member will continue to hold on to its partition if there are no other owners 2. If there are 1+ owners to the same partition. One with the highest generation will win. 3. If two members of the same generation hold on to the same partition. We will log an error but remove both from the assignment. (Same with the current logic) Here are some important notes that lead to the patch: - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown. - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error. This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive To summarize the current behavior of different errors: `REBALANCE_IN_PROGRESS` - heartbeat: requestRejoin if member state is stable - joinGroup: rejoin immediately - syncGroup: rejoin immediately - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress. `UNKNOWN_MEMBER_ID` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately - syncGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately `ILLEGAL_GENERATION` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing. then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately Reviewers: David Jacot <djacot@confluent.io> commit 10b3e667132934084a2d275a204a1a782c2df94e Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri Apr 28 10:32:11 2023 +0200 KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d796480fe87fd819fc0ac560ca318759180d4644 Author: Luke Chen <showuon@gmail.com> Date: Fri Apr 28 14:35:12 2023 +0800 KAFKA-14909: check zkMigrationReady tag before migration (#13631) 1. add ZkMigrationReady in apiVersionsResponse 2. check all nodes if ZkMigrationReady are ready before moving to next migration state Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com> commit c708f7ba5f4f449920cec57a5b69e84e92128b54 Author: Colin Patrick McCabe <cmccabe@apache.org> Date: Thu Apr 27 19:15:26 2023 -0700 MINOR: remove spurious call to fatalFaultHandler (#13651) Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805. We should only invoke the fatal fault handller if we are unable to generate the activation records. If we are unable to write the activation records, a controller failover should be sufficient to remedy the situation. Co-authored-by: Luke Chen showuon@gmail.com Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com> commit 056657d84d84e116ffc9460872945b4d2b479ff3 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 17:04:56 2023 -0400 MINOR add license to reviewers.py commit a08f31ecfefca1a51d64137a344e62a236740e62 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 14:32:19 2023 -0400 MINOR: Adding reviewers.py (#11096) This script can be used to help build the "Reviewers: " string we include in commit messages. Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit 70493336171363cfa95237a0fe14ef57090553e4 Author: Colin P. McCabe <cmccabe@apache.org> Date: Wed Apr 26 16:10:46 2023 -0700 KAFKA-14943: Fix ClientQuotaControlManager validation Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of internal ZK representation details that treated them both as "client configs.") Add unit tests for ClientQuotaControlManager.isValidIpEntity and ClientQuotaControlManager.configKeysForEntityType. This change doesn't affect metadata record application, only input validation. If there are bad client quotas that are set currently, this change will not alter the current behavior (of throwing an exception and ignoring the bad quota). commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051 Author: David Jacot <djacot@confluent.io> Date: Thu Apr 27 14:05:41 2023 +0200 KAFKA-14462; [9/N] Add RecordHelpers (#13544) This patch adds RecordHelpers. Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9 Author: LinShunKang <linshunkang.chn@gmail.com> Date: Thu Apr 27 10:44:08 2023 +0800 KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545) This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035 Direct use ByteBuffer instead of byte[] to deserialize. Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro> commit c1b5c75d9271638776392822a094e9e7ef37f490 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 26 10:20:30 2023 -0400 KAFKA-14805 KRaft controller supports pre-migration mode (#13407) This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode, the controller will only allow certain write operations. The purpose of this is to disallow metadata changes when the controller is waiting for the ZK migration records to be committed. The following ControllerWriteEvent operations are permitted in pre-migration mode * completeActivation * maybeFenceReplicas * writeNoOpRecord * processBrokerHeartbeat * registerBroker (only for migrating ZK brokers) * unregisterBroker Raft events and other controller events do not follow the same code path as ControllerWriteEvent, so they are not affected by this new behavior. This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean MIGRATION rather than PRE_MIGRATION. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 007c0d375a6e70aefb65f58f9f096016c4cbea16 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 26 17:53:56 2023 +0530 KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634) Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit baf127a6633161cb52747467880b006d2f54d3bd Author: Greg Harris <greg.harris@aiven.io> Date: Wed Apr 26 00:30:13 2023 -0700 KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429) Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io> commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef Author: Victoria Xia <victoria.xia@confluent.io> Date: Tue Apr 25 22:39:23 2023 -0400 KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622) Reviewers: Matthias J. Sax <matthias@confluent.io> commit a7d0b3f753708a93aea92e614833f6f6e7443234 Author: Said Boudjelda <bmscomp@gmail.com> Date: Tue Apr 25 23:31:04 2023 +0200 MINOR: Upgrade gradle to 8.1.1 (#13625) Also upgrade gradle plugins: - `org.owasp.dependencycheck` gradle plugin to version `8.2.1` - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1` Gradle release notes: * https://docs.gradle.org/8.1.1/release-notes.html Reviewers: Ismael Juma <ismael@juma.me.uk> commit 9a36da12b7359b7158332c541655716312efb5b3 Author: David Jacot <djacot@confluent.io> Date: Tue Apr 25 18:50:51 2023 +0200 KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538) This patch adds ConsumerGroupMember. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io> commit 4780dc773f2cd5a20fa5be38d20137745690a888 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue Apr 25 21:20:35 2023 +0530 KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4 Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Tue Apr 25 13:28:37 2023 +0100 KAFKA-14592: Move FeatureCommand to tools (#13459) KAFKA-14592: Move FeatureCommand to tools Reviewers: Luke Chen <showuon@gmail.com> commit d83a734c41c43a03aa571e602cf8535f0893fd79 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 25 12:02:24 2023 +0200 MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626) The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call. Reviewers: maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com> commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 15:29:57 2023 -0700 KAFKA-12446: update change encoding to use varint (#13533) KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io> commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402 Author: Victoria Xia <victoria.xia@confluent.io> Date: Mon Apr 24 17:06:26 2023 -0400 KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615) Reviewers: Matthias J. Sax <matthias@confluent.io> commit 6dcdb017327587a6943fa868595fa3488c7f7ef7 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 12:40:25 2023 -0700 KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592) Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks. This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io> commit 2271e748a11919d07698ebce759dca2e3075596a Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Mon Apr 24 17:21:19 2023 +0800 MINOR: fix zookeeper_migration_test.py (#13620) Reviewers: Mickael Maison <mimaison@users.noreply.github.com> commit c3241236296f9aaee103e475f242e32f04d4c256 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon Apr 24 14:06:20 2023 +0530 KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 7061475445cb6314e7cf4f9848384224b14f4395 Author: Greg Harris <greg.harris@aiven.io> Date: Fri Apr 21 12:55:41 2023 -0700 KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575) Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest` Reviewers: Josep Prat <jlprat@apache.org> commit ecdef88f744410039b88e90a5979078d5735aa06 Author: Matthias J. Sax <matthias@confluent.io> Date: Fri Apr 21 12:48:05 2023 -0700 MINOR: updated KS release notes for 3.5 (#13577) Reviewers: Walker Carlson <wcarlson@confluent.io> commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Fri Apr 21 15:02:06 2023 +0200 MINOR: fix noticed typo in raft and metadata projects (#13612) Reviewers: Josep Prat <jlprat@apache.org> commit c39bf714bbaf2a632be4c2a7e446553fe40ba129 Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:22:16 2023 +0200 KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537) This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 2d0b816150c79057c813387bd126523b6326a1fc Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:19:04 2023 +0200 MINOR: Move `ControllerPurgatory` to `server-common` (#13555) This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit df137752542c005c6998c37c03222ffbeca0f349 Author: Purshotam Chauhan <pchauhan@confluent.io> Date: Fri Apr 21 14:08:23 2023 +0530 KAFKA-14828: Remove R/W locks using persistent data structures (#13437) Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only. In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation. Reviewers: Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Divij Vaidya <diviv@amazon.com> commit 2ee770ac7e576c85a911ccb307339be3d58a8942 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu Apr 20 10:21:26 2023 -0700 Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)" This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc. commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3 Author: Justine Olshan <jolshan@confluent.io> Date: Thu Apr 20 09:30:11 2023 -0700 KAFKA-14904: Pending state blocked verification of transactions (#13579) KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction. There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes. When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions. The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely. This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed. Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit 7f175feacaba4187bbb3631dff3a1330060f191e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Thu Apr 20 13:50:29 2023 +0530 KAFKA-14586: Adding redirection for StreamsResetter (#13614) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com> commit e14dd8024adae746c50c8b7d9cd268e859669576 Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com> Date: Thu Apr 20 05:29:27 2023 +0200 KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432) We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic: 1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry. 2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of the client. 3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`. We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them. Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io> commit f5de4daa71f0ad31aa64443c5ff43c59712feea5 Author: Ron Dagostino <rndgstn@gmail.com> Date: Wed Apr 19 19:45:26 2023 -0400 KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change events. However, it does not distinguish between issues related to feature changes actually failing and other exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues. We shutdown the broker only for FeatureCacheUpdateException. Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 11c8bf4826197533807b2132cfc6599ba70de1c1 Author: Victoria Xia <victoria.xia@confluent.io> Date: Wed Apr 19 19:34:36 2023 -0400 KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609) This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter. Part of KIP-914. Reviewer: Matthias J. Sax <matthias@confluent.io> commit 809966a9a06664e0b521c2298fa0de834e443607 Author: Matthew de Detrich <matthew.dedetrich@aiven.io> Date: Wed Apr 19 20:54:07 2023 +0200 KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478) Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address. Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org> commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 19 14:19:13 2023 -0400 KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606) This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b10716e72370c4e128bddb17bcc107ccab221e47 Author: hudeqi <1217150961@qq.com> Date: Thu Apr 20 00:49:08 2023 +0800 KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com> commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc Author: Keith Wall <kwall@redhat.com> Date: Wed Apr 19 03:58:29 2023 +0100 KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572) Changes SocketServer to set the setReuseAddress(true) socket option. This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test. Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com> commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 19 00:46:03 2023 +0530 MINOR: Fixing gradle build during compileScala and compileTestScala (#13588) Reviewers: Chia-Ping Tsai <chia7712@gmail.com> commit 3388adf1b52f545e2d69edd30cdd53241b2887f3 Author: Matthias J. Sax <matthias@confluent.io> Date: Tue Apr 18 11:32:27 2023 -0700 MINOR: rename internal FK-join processor classes (#13589) Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io> commit abca86511ecc9e081d676976ff6d3b845308f444 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Tue Apr 18 12:41:38 2023 -0400 KAFKA-14881: Rework UserScramCredentialRecord (#13513) Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This is necessary to support migration from ZK, since those are the fields we stored in ZK. Update latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version. Moved ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more appropriate. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 61530d68ce83467de6190a52da37b3c0af84f0ef Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 09:37:04 2023 -0400 KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526) This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields. Reviewers: David Jacot <djacot@confluent.io> commit b36a170aa3b2738177d7229859db765e0115385b Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 18 13:36:56 2023 +0200 MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593) Reviewers: Luke Chen <showuon@gmail.com> commit 5e9d4de748dff7b91043a9b799716ab4becdae7e Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 04:41:54 2023 -0400 KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511) This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io> commit 454b72161a76b1687a1263157d7cc30a1bdb2506 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Tue Apr 18 09:40:14 2023 +0200 KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557) KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow. To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread. commit e27926f92b1f6b34ed6731f33c712a5d0d594275 Author: Ron Dagostino <rndgstn@gmail.com> Date: Mon Apr 17 17:52:28 2023 -0400 KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io> commit 7159f6c1a81ead277030f55f293943270346ad4e Author: Alyssa Huang <ahuang@confluent.io> Date: Mon Apr 17 11:03:45 2023 -0700 MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it should have been. Add a regression test. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit cc88822986e4004e772f376eae47dbe08e29d137 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 10 16:29:27 2023 -0700 small changes commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 25 14:58:20 2023 -0700 grammar and code comments changes commit 5550e4934ade28f069e4ae9ab97e76982ff3401f Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 17:39:48 2023 -0700 fixed formatting stuff commit 28ab29819bf327309ce07196a3fb0492f7fb2281 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 11:38:57 2023 -0700 consumers -> members commit e56ded267456bbe1c22ded0d037953545b921d1d Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 11:26:23 2023 -0700 Interface changes incorporated commit 2d50c278c18baa11b0235330149dd8f9def57c3d Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 13 14:47:33 2023 -0700 Renamed topicIdToPartition class commit d33bbfac053ccac77204ac4253698ed356d7c621 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 17:31:26 2023 -0700 Added stickiness test commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 13:28:24 2023 -0700 Separated uniform and general builder in different files, added more tests commit ef814867aba68fc4813359c5cef6ae155ed49a60 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:24:27 2023 -0700 import changes commit dffad677c6e4a33625401d621f046af6f21feb99 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:16:33 2023 -0700 Fixed checkstyle and added test file commit 7fd779418df3504a490c0d10f3848360a4417077 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 5 15:13:38 2023 -0700 First draft optimised uniform assignor code commit 309cc6141f7962e794c90d4107f460fafd51697d Merge: 3d9264547c 2c1cf03a89 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Mon Apr 17 11:03:07 2023 -0700 Merge branch 'trunk' into rreddy-22/KAFKA-14514 commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 10:58:16 2023 -0700 Interface changes incorporated, added stickiness test and non-existent topic test. commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436 Merge: a1bdb58e08 57b94ca208 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 09:49:23 2023 -0700 Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514 # Conflicts: # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java commit a1bdb58e082d98b73f50860db5b35684f4eba51b Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 20:01:59 2023 -0700 Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc. commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 11:28:40 2023 -0700 Addressed some PR comments commit 918d262ae52892c13766649db847ede6928921f8 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7369db3744b4c89f625f6b73db331eaf0512c4eb Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit a925b02dd3d78c8b9c1415c083f0544edcb404fe Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit 48a982232c6f1ff371cb44333fd12d6b37ef381a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 13:00:07 2023 -0700 Removed topicIdToPartition class, changed attribute names and added java doc for getter methods commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 15:09:44 2023 -0700 Made all attributes private and added getter methods, changed names according to PR comments commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:25:33 2023 -0700 Added toString method and hash commit 5065109332845541c5591a52683f21d37982d80a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 13:43:26 2023 -0700 Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure commit 57b94ca208cb374b22d61eb044ef8004ea09479f Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7352cbb5683f676382f56e6ae5c926f3cae98824 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0 Merge: 4b321aa537 637bc92ba1 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Thu Apr 6 10:37:58 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514 commit 4b321aa5374427a15346be9b87b7c545ccb7db61 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 74105291cc687fe41b75de692fc624ab1335341a Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit 3df34605e078beb9babcbeadc7b119a70da63796 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation

commit 1c955b5e251aedc174cf28f43037f08162d89b49 Merge: 2d40125fa8 81f1ccd7a2 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 13:02:54 2023 -0700 Merge branch 'rreddy-22/Rack-Assignor-Interface-Changes' of github.com:rreddy-22/kafka-rreddy into rreddy-22/Rack-Assignor-Interface-Changes commit 81f1ccd7a25d85eaf1cff713df402e0100d4b4a3 Merge: a39e76f63f 84691b11f6 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Mon Jul 24 12:56:40 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes commit 2d40125fa8ba16b4f07e6bfe410ceacfd8a112e8 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 11:45:21 2023 -0700 minor commit a39e76f63f312ecacdcebd2c5015777e160ca0b3 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 10:04:10 2023 -0700 reverted reviewers.py changes commit c0b464c8936f1df4f6540b8abe7c1fa88fce6e81 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 00:02:03 2023 -0700 reverted grammar changes commit 6fe72dfce921ab9d77ba6b28d4104c5d92c6dbf3 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 23:56:35 2023 -0700 minor commit a30a53463e55b72699617360d1c888b7ee310336 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 23:55:35 2023 -0700 minor commit 57b9a6233a4f697f285b85e6394d3dd407f16404 Merge: 061dac797b 4981fa939d Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Sun Jul 23 19:52:07 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes commit 061dac797bae56d91659fbc8d7900117eb10a866 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:43:12 2023 -0700 Moved SubscribedTopicMetadata to the consumer package commit 6650ea5cd707f03713fc2c69ba76fafd56d886b8 Merge: 22f2df02de faafed25a1 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:33:17 2023 -0700 Merge remote-tracking branch 'origin/rreddy-22/Rack-Assignor-Interface-Changes' into rreddy-22/Rack-Assignor-Interface-Changes commit 22f2df02deabe1a6b5b8b8cceb432486b9ef4470 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:25:11 2023 -0700 minor changes commit faafed25a10d7ff79fb8a24ab62aa222c0e6516a Merge: 4511683d4e 1bf73d89d0 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Fri Jul 21 12:50:59 2023 -0700 Merge branch 'trunk' into rreddy-22/Rack-Assignor-Interface-Changes commit 4511683d4ef489cb979769ff6775dfe0250cdeeb Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 21 03:09:27 2023 -0700 Changes based on PR comments commit 579bb948fcc78afd163410a5bc43832bc87aaf71 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Jul 19 14:45:18 2023 -0700 New interfaces and classes added to facilitate passing rack information to the assignor, modified tests to incorporate changes commit f68471a99f2e0260b36047a7596b0f9de48794ad Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Jul 13 11:55:20 2023 -0700 Small edits commit c50f3bdbf79f1ca127355d2a9002b1fdbf0baa2f Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 11 16:30:32 2023 -0700 Removed extra lines commit 478488e0943aa879b27eec5ae5eb79a23dd9cab3 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 11 16:21:54 2023 -0700 Added rack interface changes, added a new TopicAndClusterMetadata class to handle topic and cluster images and added an AbstractPartitionAssignor commit ce540f943b8dae0786a87e51cafacb66efd1e64d Merge: 9815d6db04 fd5b300b57 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:45:27 2023 -0600 Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized commit 9815d6db04c264456b7ead7fc5f735513a868c7d Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:23:00 2023 -0600 Abstract Partition Assignor commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Jun 29 16:36:53 2023 -0700 Refactored code to reduce format conversion time commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 31 16:21:01 2023 -0700 test changes and code changes commit e1a032579129a2cd3aae1e93f46570bd57d3343c Author: Ritika Reddy <rreddy@confluent.io> Date: Tue May 16 09:28:57 2023 -0700 small changes commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu May 11 20:36:59 2023 -0700 Squashed commit of the following: commit c757af5f7c630d532bfee5f6dc45aec603ad8a29 Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri May 12 04:39:12 2023 +0200 KAFKA-14752: Kafka examples improvements - demo changes (#13517) KAFKA-14752: Kafka examples improvements - demo changes Reviewers: Luke Chen <showuon@gmail.com> commit 54a4067f81e1434d956ef797274f7b437fe49ea1 Author: Kamal Chandraprakash <kchandraprakash@uber.com> Date: Thu May 11 21:49:21 2023 +0530 KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060) Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org> commit bd65db82b4bad623b0bb31398979e466978148da Author: Josep Prat <josep.prat@aiven.io> Date: Thu May 11 17:55:26 2023 +0200 MINOR: clean up unused methods in core utils (#13706) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org> commit 38adb3956979d7025d148612975cc0b82200b2e1 Author: Jeff Kim <kimkb2011@gmail.com> Date: Thu May 11 07:51:41 2023 -0400 MINOR: add test tag for testDeadToDeadIllegalTransition (#13694) Reviewers: David Jacot <djacot@confluent.io> commit ee4132863553fb4fd2df715b2fbd77f349f978b8 Author: Federico Valeri <fedevaleri@gmail.com> Date: Thu May 11 11:19:32 2023 +0200 KAFKA-14752: Kafka examples improvements - processor changes (#13516) Reviewers: Luke Chen <showuon@gmail.com> commit a263627adb75f1ca5c87f1482cc70b994ba49d63 Author: Mickael Maison <mimaison@users.noreply.github.com> Date: Thu May 11 11:02:45 2023 +0200 MINOR: Remove unused methods in CoreUtils (#13170) Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com> commit 920a3601ffa7266edf12f3559cfe97c8a5929d03 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Thu May 11 04:13:29 2023 +0200 MINOR: fix a small typo in SharedServer.scala (#13693) Diabled -> Disabled commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc Author: A. Sophie Blee-Goldman <sophie@confluent.io> Date: Wed May 10 13:39:15 2023 -0700 HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695) The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen() Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io> commit f17fb75b2de32512f14cb94a7d1bfb0f37485780 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Wed May 10 16:41:52 2023 +0200 KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690) Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 4653507926a42dccda5c086fcae6278afcfc53ca Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Wed May 10 05:09:12 2023 -0700 KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443) This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers. Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io> commit 625ef176ee5f167786003c2d88498632b0b7014b Author: Luke Chen <showuon@gmail.com> Date: Wed May 10 16:40:20 2023 +0800 MINOR: remove kraft readme link (#13691) The config/kraft/README.md is already removed. We should also remove the link. Reviewers: dengziming <dengziming1993@gmail.com> commit 228434d23583189cdcaa7f4a90ebb178ccc17c73 Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue May 9 10:49:27 2023 -0400 KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644) This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java. Reviewers: David Jacot <djacot@confluent.io> commit 59ba9dbbc927ddc8660d0d98d9422909fd306758 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 9 17:58:45 2023 +0530 KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688) `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods. This refactoring changes no functionality other than restoring the older methods. Reviewers: Randall Hauch <rhauch@gmail.com> commit b40a7fc037bb1543c3355fad9c71570f770f5177 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon May 8 14:24:11 2023 -0700 HOTFIX: fix broken Streams upgrade system test (#13654) Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io> commit 7634eee2627da39937e3112ffc58bd7cfedc98f2 Author: David Jacot <djacot@confluent.io> Date: Mon May 8 20:46:07 2023 +0200 KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638) This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state. Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 86daf8ce6573eb79d6e78381dbab738055f914c4 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Mon May 8 20:09:47 2023 +0530 KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594) #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR. Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places. --------- Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6 Author: Christo Lolov <lolovc@amazon.com> Date: Mon May 8 15:24:52 2023 +0100 KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621) Migrates ChangeLogReader mock in TaskManagerTest to mockito. Reviewer: Bruno Cadonna <cadonna@apache.org> commit 347238948b86882a47faee4a2916d1b01333d95f Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Mon May 8 07:36:36 2023 +0100 KAFKA-14662: Update the ACL list in the doc (#13660) Added the missing ACLs to the doc. Reviewers: Luke Chen <showuon@gmail.com> commit a556def5724efb1dc96bd2d389411a8d2f802f53 Author: Divij Vaidya <diviv@amazon.com> Date: Mon May 8 08:35:14 2023 +0200 MINOR: Print the cause of failure for test (#13672) Motivation PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build): org.opentest4j.AssertionFailedError: expected: <false> but was: <true> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31) at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228) at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is. Change 1. Make the test a bit more robust by waiting for server startup. 2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure. Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com> commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 08:33:31 2023 +0200 MINOR: Fix producer Callback comment (#13669) Fixes the wrong exception name: OffsetMetadataTooLargeException. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com> commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399 Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 04:15:52 2023 +0200 KAFKA-14752: Kafka examples improvements - producer changes (#13515) KAFKA-14752: Kafka examples improvements - producer changes Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com> commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714 Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Sat May 6 02:56:26 2023 +0800 MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659) Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro> commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 14:05:20 2023 +0200 KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312) Motivation Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit. In this PR, we only update varInt32 encoding/decoding. Changes This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation. Results Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com> commit e34f88403159cc8381da23dafdf7e3d7403114a2 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 13:55:17 2023 +0200 KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623) When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io> commit 0822ce0ed1a106a510930bc9ac53a266f54684d7 Author: David Arthur <mumrah@gmail.com> Date: Fri May 5 04:35:26 2023 -0400 KAFKA-14840: Support for snapshots during ZK migration (#13461) This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active controller loaded a snapshot, it would get out of sync with the ZK state. In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass and without using too much memory. Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com> commit 97c36f3f3142580325daa1a6aadb662893390561 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 12:20:33 2023 -0700 HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540 commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 11:18:03 2023 -0700 MINOR: improve QuorumController logging #13540 When creating the QuorumController, log whether ZK migration is enabled. When applying a feature level record which sets the metadata version, log the metadata version enum rather than the numeric feature level. Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the end of replay. When TRACE is enabled, log every record that is replayed in QuorumController. Since some records may contain sensitive information, create RecordRedactor to assist in logging only what is safe to put in the log4j file. Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged at INFO, and additions are logged at TRACE. Remove SnapshotReason.java, SnapshotReasonTest.java, and QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to org.apache.kafka.image.publisher.SnapshotGenerator. Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff Author: Justine Olshan <jolshan@confluent.io> Date: Thu May 4 09:55:45 2023 -0700 KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607) Also modifies verification to only add a partition to verify if it is transactional. When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different. Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit ea81e99e5980c807414651034a8c60426a158ca4 Author: Philip Nee <pnee@confluent.io> Date: Thu May 4 09:20:01 2023 -0700 KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149) Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer. This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin. Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest. Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dc7819d7f1fe6b0160cd95246420ab10c335410b Author: Christo Lolov <lolovc@amazon.com> Date: Thu May 4 11:00:33 2023 +0100 KAFKA-14594: Move LogDirsCommand to tools module (#13122) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d46c3f259cce25c43f20fba3943d5cb34ed909ea Author: David Mao <47232755+splett2@users.noreply.github.com> Date: Wed May 3 17:09:43 2023 -0700 MINOR: Reduce number of threads created for integration test brokers (#13655) The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker. Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io> commit c08120f83f7318f15dcf14d525876d18caf6afd0 Author: Jason Gustafson <jason@confluent.io> Date: Wed May 3 15:25:32 2023 -0700 MINOR: Allow tagged fields with version subset of flexible version range (#13551) The generated message types are missing a range check for the case when the tagged version range is a subset of the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the number of tags serialized. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b620c03ccf48d6d92b219cba35bb1e5e248d2547 Author: Luke Chen <showuon@gmail.com> Date: Thu May 4 01:08:25 2023 +0800 KAFKA-14946: fix NPE when merging the deltatable (#13653) Fix NPE while merging the deltatable. Because it's possible that hashTier is not null but deltatable is null (ex: removing data), we should have null check while merging for deltatable like other places did. Also added tests that will fail without this change. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 4a0b6ebf60ed7614f042443460b490971e8662a4 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 2 23:16:46 2023 +0530 KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657) Reviewers: Chris Egerton <chrise@aiven.io> commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e Author: David Jacot <djacot@confluent.io> Date: Tue May 2 18:04:50 2023 +0200 KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637) This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group. Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949 Author: Christo Lolov <lolovc@amazon.com> Date: Tue May 2 16:39:31 2023 +0100 MINOR: Remove unnecessary code in client/connect (#13259) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3 Author: Federico Valeri <fedevaleri@gmail.com> Date: Tue May 2 14:28:42 2023 +0200 KAFKA-14752: Kafka examples improvements - consumer changes (#13514) KAFKA-14752: Kafka examples improvements - consumer changes This is extracted from the original PR for better review. https://github.com/apache/kafka/pull/13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com> Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com> commit 141c76a2c904705f2cd484e96767fcb217c5db25 Author: Bruno Cadonna <cadonna@apache.org> Date: Tue May 2 14:00:34 2023 +0200 KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529) 1. Migrates topology builder mock in TaskManagerTest to mockito. 2. Replaces the unit test to verify if subscribed partitions are added to topology metadata. 3. Modifies signatures of methods for adding subscribed partitions to topology metadata to use sets instead of lists. This makes the intent of the methods clearer and makes the tests more portable. Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org> commit 21af1918eafa30812a955c3c0295b9e968841cd3 Author: Luke Chen <showuon@gmail.com> Date: Tue May 2 09:54:12 2023 +0800 MINOR: Add reason to exceptions in QuorumController (#13648) Saw this error message in log: ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController) The null reason is not helpful with only RejectedExecutionException. Adding the reason to it. Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit 4773961a44d1f0d1e11d662c7e0fc955027bced2 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon May 1 21:51:09 2023 +0530 MINOR: Fix Javadoc for configureAdminResources in Connect's RestServer (#13635) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io> commit e29942347acc70aa85d47e84e2021f9c24cd7c80 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Mon May 1 09:56:04 2023 -0400 KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628) Handle migrating SCRAM records in ZK when migrating from ZK to KRaft. This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes. Reviewers: David Arthur <mumrah@gmail.com> commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 08:46:40 2023 -0700 MINOR: Fixing typos in the ConsumerCoordinator (#13618) Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io> commit e55fbceb6667cc1455f7fb2e96421c85741fa7df Author: Anton Agestam <anton.agestam@aiven.io> Date: Fri Apr 28 11:54:28 2023 +0100 MINOR: Fix incorrect description of SessionLifetimeMs (#13649) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984 Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 02:08:32 2023 -0700 KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550) This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS. This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method. In short, it should no long check and validate only the highest generation. Instead, we consider 3 cases: 1. Member will continue to hold on to its partition if there are no other owners 2. If there are 1+ owners to the same partition. One with the highest generation will win. 3. If two members of the same generation hold on to the same partition. We will log an error but remove both from the assignment. (Same with the current logic) Here are some important notes that lead to the patch: - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown. - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error. This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive To summarize the current behavior of different errors: `REBALANCE_IN_PROGRESS` - heartbeat: requestRejoin if member state is stable - joinGroup: rejoin immediately - syncGroup: rejoin immediately - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress. `UNKNOWN_MEMBER_ID` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately - syncGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately `ILLEGAL_GENERATION` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing. then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately Reviewers: David Jacot <djacot@confluent.io> commit 10b3e667132934084a2d275a204a1a782c2df94e Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri Apr 28 10:32:11 2023 +0200 KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d796480fe87fd819fc0ac560ca318759180d4644 Author: Luke Chen <showuon@gmail.com> Date: Fri Apr 28 14:35:12 2023 +0800 KAFKA-14909: check zkMigrationReady tag before migration (#13631) 1. add ZkMigrationReady in apiVersionsResponse 2. check all nodes if ZkMigrationReady are ready before moving to next migration state Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com> commit c708f7ba5f4f449920cec57a5b69e84e92128b54 Author: Colin Patrick McCabe <cmccabe@apache.org> Date: Thu Apr 27 19:15:26 2023 -0700 MINOR: remove spurious call to fatalFaultHandler (#13651) Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805. We should only invoke the fatal fault handller if we are unable to generate the activation records. If we are unable to write the activation records, a controller failover should be sufficient to remedy the situation. Co-authored-by: Luke Chen showuon@gmail.com Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com> commit 056657d84d84e116ffc9460872945b4d2b479ff3 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 17:04:56 2023 -0400 MINOR add license to reviewers.py commit a08f31ecfefca1a51d64137a344e62a236740e62 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 14:32:19 2023 -0400 MINOR: Adding reviewers.py (#11096) This script can be used to help build the "Reviewers: " string we include in commit messages. Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit 70493336171363cfa95237a0fe14ef57090553e4 Author: Colin P. McCabe <cmccabe@apache.org> Date: Wed Apr 26 16:10:46 2023 -0700 KAFKA-14943: Fix ClientQuotaControlManager validation Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of internal ZK representation details that treated them both as "client configs.") Add unit tests for ClientQuotaControlManager.isValidIpEntity and ClientQuotaControlManager.configKeysForEntityType. This change doesn't affect metadata record application, only input validation. If there are bad client quotas that are set currently, this change will not alter the current behavior (of throwing an exception and ignoring the bad quota). commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051 Author: David Jacot <djacot@confluent.io> Date: Thu Apr 27 14:05:41 2023 +0200 KAFKA-14462; [9/N] Add RecordHelpers (#13544) This patch adds RecordHelpers. Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9 Author: LinShunKang <linshunkang.chn@gmail.com> Date: Thu Apr 27 10:44:08 2023 +0800 KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545) This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035 Direct use ByteBuffer instead of byte[] to deserialize. Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro> commit c1b5c75d9271638776392822a094e9e7ef37f490 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 26 10:20:30 2023 -0400 KAFKA-14805 KRaft controller supports pre-migration mode (#13407) This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode, the controller will only allow certain write operations. The purpose of this is to disallow metadata changes when the controller is waiting for the ZK migration records to be committed. The following ControllerWriteEvent operations are permitted in pre-migration mode * completeActivation * maybeFenceReplicas * writeNoOpRecord * processBrokerHeartbeat * registerBroker (only for migrating ZK brokers) * unregisterBroker Raft events and other controller events do not follow the same code path as ControllerWriteEvent, so they are not affected by this new behavior. This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean MIGRATION rather than PRE_MIGRATION. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 007c0d375a6e70aefb65f58f9f096016c4cbea16 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 26 17:53:56 2023 +0530 KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634) Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit baf127a6633161cb52747467880b006d2f54d3bd Author: Greg Harris <greg.harris@aiven.io> Date: Wed Apr 26 00:30:13 2023 -0700 KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429) Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io> commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef Author: Victoria Xia <victoria.xia@confluent.io> Date: Tue Apr 25 22:39:23 2023 -0400 KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622) Reviewers: Matthias J. Sax <matthias@confluent.io> commit a7d0b3f753708a93aea92e614833f6f6e7443234 Author: Said Boudjelda <bmscomp@gmail.com> Date: Tue Apr 25 23:31:04 2023 +0200 MINOR: Upgrade gradle to 8.1.1 (#13625) Also upgrade gradle plugins: - `org.owasp.dependencycheck` gradle plugin to version `8.2.1` - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1` Gradle release notes: * https://docs.gradle.org/8.1.1/release-notes.html Reviewers: Ismael Juma <ismael@juma.me.uk> commit 9a36da12b7359b7158332c541655716312efb5b3 Author: David Jacot <djacot@confluent.io> Date: Tue Apr 25 18:50:51 2023 +0200 KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538) This patch adds ConsumerGroupMember. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io> commit 4780dc773f2cd5a20fa5be38d20137745690a888 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue Apr 25 21:20:35 2023 +0530 KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4 Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Tue Apr 25 13:28:37 2023 +0100 KAFKA-14592: Move FeatureCommand to tools (#13459) KAFKA-14592: Move FeatureCommand to tools Reviewers: Luke Chen <showuon@gmail.com> commit d83a734c41c43a03aa571e602cf8535f0893fd79 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 25 12:02:24 2023 +0200 MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626) The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call. Reviewers: maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com> commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 15:29:57 2023 -0700 KAFKA-12446: update change encoding to use varint (#13533) KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io> commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402 Author: Victoria Xia <victoria.xia@confluent.io> Date: Mon Apr 24 17:06:26 2023 -0400 KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615) Reviewers: Matthias J. Sax <matthias@confluent.io> commit 6dcdb017327587a6943fa868595fa3488c7f7ef7 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 12:40:25 2023 -0700 KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592) Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks. This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io> commit 2271e748a11919d07698ebce759dca2e3075596a Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Mon Apr 24 17:21:19 2023 +0800 MINOR: fix zookeeper_migration_test.py (#13620) Reviewers: Mickael Maison <mimaison@users.noreply.github.com> commit c3241236296f9aaee103e475f242e32f04d4c256 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon Apr 24 14:06:20 2023 +0530 KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 7061475445cb6314e7cf4f9848384224b14f4395 Author: Greg Harris <greg.harris@aiven.io> Date: Fri Apr 21 12:55:41 2023 -0700 KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575) Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest` Reviewers: Josep Prat <jlprat@apache.org> commit ecdef88f744410039b88e90a5979078d5735aa06 Author: Matthias J. Sax <matthias@confluent.io> Date: Fri Apr 21 12:48:05 2023 -0700 MINOR: updated KS release notes for 3.5 (#13577) Reviewers: Walker Carlson <wcarlson@confluent.io> commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Fri Apr 21 15:02:06 2023 +0200 MINOR: fix noticed typo in raft and metadata projects (#13612) Reviewers: Josep Prat <jlprat@apache.org> commit c39bf714bbaf2a632be4c2a7e446553fe40ba129 Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:22:16 2023 +0200 KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537) This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 2d0b816150c79057c813387bd126523b6326a1fc Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:19:04 2023 +0200 MINOR: Move `ControllerPurgatory` to `server-common` (#13555) This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit df137752542c005c6998c37c03222ffbeca0f349 Author: Purshotam Chauhan <pchauhan@confluent.io> Date: Fri Apr 21 14:08:23 2023 +0530 KAFKA-14828: Remove R/W locks using persistent data structures (#13437) Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only. In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation. Reviewers: Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Divij Vaidya <diviv@amazon.com> commit 2ee770ac7e576c85a911ccb307339be3d58a8942 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu Apr 20 10:21:26 2023 -0700 Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)" This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc. commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3 Author: Justine Olshan <jolshan@confluent.io> Date: Thu Apr 20 09:30:11 2023 -0700 KAFKA-14904: Pending state blocked verification of transactions (#13579) KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction. There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes. When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions. The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely. This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed. Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit 7f175feacaba4187bbb3631dff3a1330060f191e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Thu Apr 20 13:50:29 2023 +0530 KAFKA-14586: Adding redirection for StreamsResetter (#13614) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com> commit e14dd8024adae746c50c8b7d9cd268e859669576 Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com> Date: Thu Apr 20 05:29:27 2023 +0200 KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432) We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic: 1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry. 2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of the client. 3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`. We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them. Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io> commit f5de4daa71f0ad31aa64443c5ff43c59712feea5 Author: Ron Dagostino <rndgstn@gmail.com> Date: Wed Apr 19 19:45:26 2023 -0400 KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change events. However, it does not distinguish between issues related to feature changes actually failing and other exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues. We shutdown the broker only for FeatureCacheUpdateException. Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 11c8bf4826197533807b2132cfc6599ba70de1c1 Author: Victoria Xia <victoria.xia@confluent.io> Date: Wed Apr 19 19:34:36 2023 -0400 KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609) This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter. Part of KIP-914. Reviewer: Matthias J. Sax <matthias@confluent.io> commit 809966a9a06664e0b521c2298fa0de834e443607 Author: Matthew de Detrich <matthew.dedetrich@aiven.io> Date: Wed Apr 19 20:54:07 2023 +0200 KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478) Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address. Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org> commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 19 14:19:13 2023 -0400 KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606) This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b10716e72370c4e128bddb17bcc107ccab221e47 Author: hudeqi <1217150961@qq.com> Date: Thu Apr 20 00:49:08 2023 +0800 KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com> commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc Author: Keith Wall <kwall@redhat.com> Date: Wed Apr 19 03:58:29 2023 +0100 KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572) Changes SocketServer to set the setReuseAddress(true) socket option. This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test. Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com> commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 19 00:46:03 2023 +0530 MINOR: Fixing gradle build during compileScala and compileTestScala (#13588) Reviewers: Chia-Ping Tsai <chia7712@gmail.com> commit 3388adf1b52f545e2d69edd30cdd53241b2887f3 Author: Matthias J. Sax <matthias@confluent.io> Date: Tue Apr 18 11:32:27 2023 -0700 MINOR: rename internal FK-join processor classes (#13589) Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io> commit abca86511ecc9e081d676976ff6d3b845308f444 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Tue Apr 18 12:41:38 2023 -0400 KAFKA-14881: Rework UserScramCredentialRecord (#13513) Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This is necessary to support migration from ZK, since those are the fields we stored in ZK. Update latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version. Moved ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more appropriate. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 61530d68ce83467de6190a52da37b3c0af84f0ef Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 09:37:04 2023 -0400 KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526) This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields. Reviewers: David Jacot <djacot@confluent.io> commit b36a170aa3b2738177d7229859db765e0115385b Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 18 13:36:56 2023 +0200 MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593) Reviewers: Luke Chen <showuon@gmail.com> commit 5e9d4de748dff7b91043a9b799716ab4becdae7e Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 04:41:54 2023 -0400 KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511) This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io> commit 454b72161a76b1687a1263157d7cc30a1bdb2506 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Tue Apr 18 09:40:14 2023 +0200 KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557) KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow. To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread. commit e27926f92b1f6b34ed6731f33c712a5d0d594275 Author: Ron Dagostino <rndgstn@gmail.com> Date: Mon Apr 17 17:52:28 2023 -0400 KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io> commit 7159f6c1a81ead277030f55f293943270346ad4e Author: Alyssa Huang <ahuang@confluent.io> Date: Mon Apr 17 11:03:45 2023 -0700 MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it should have been. Add a regression test. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit cc88822986e4004e772f376eae47dbe08e29d137 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 10 16:29:27 2023 -0700 small changes commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 25 14:58:20 2023 -0700 grammar and code comments changes commit 5550e4934ade28f069e4ae9ab97e76982ff3401f Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 17:39:48 2023 -0700 fixed formatting stuff commit 28ab29819bf327309ce07196a3fb0492f7fb2281 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 11:38:57 2023 -0700 consumers -> members commit e56ded267456bbe1c22ded0d037953545b921d1d Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 11:26:23 2023 -0700 Interface changes incorporated commit 2d50c278c18baa11b0235330149dd8f9def57c3d Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 13 14:47:33 2023 -0700 Renamed topicIdToPartition class commit d33bbfac053ccac77204ac4253698ed356d7c621 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 17:31:26 2023 -0700 Added stickiness test commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 13:28:24 2023 -0700 Separated uniform and general builder in different files, added more tests commit ef814867aba68fc4813359c5cef6ae155ed49a60 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:24:27 2023 -0700 import changes commit dffad677c6e4a33625401d621f046af6f21feb99 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:16:33 2023 -0700 Fixed checkstyle and added test file commit 7fd779418df3504a490c0d10f3848360a4417077 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 5 15:13:38 2023 -0700 First draft optimised uniform assignor code commit 309cc6141f7962e794c90d4107f460fafd51697d Merge: 3d9264547c 2c1cf03a89 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Mon Apr 17 11:03:07 2023 -0700 Merge branch 'trunk' into rreddy-22/KAFKA-14514 commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 10:58:16 2023 -0700 Interface changes incorporated, added stickiness test and non-existent topic test. commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436 Merge: a1bdb58e08 57b94ca208 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 09:49:23 2023 -0700 Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514 # Conflicts: # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java commit a1bdb58e082d98b73f50860db5b35684f4eba51b Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 20:01:59 2023 -0700 Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc. commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 11:28:40 2023 -0700 Addressed some PR comments commit 918d262ae52892c13766649db847ede6928921f8 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7369db3744b4c89f625f6b73db331eaf0512c4eb Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit a925b02dd3d78c8b9c1415c083f0544edcb404fe Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit 48a982232c6f1ff371cb44333fd12d6b37ef381a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 13:00:07 2023 -0700 Removed topicIdToPartition class, changed attribute names and added java doc for getter methods commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 15:09:44 2023 -0700 Made all attributes private and added getter methods, changed names according to PR comments commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:25:33 2023 -0700 Added toString method and hash commit 5065109332845541c5591a52683f21d37982d80a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 13:43:26 2023 -0700 Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure commit 57b94ca208cb374b22d61eb044ef8004ea09479f Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7352cbb5683f676382f56e6ae5c926f3cae98824 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0 Merge: 4b321aa537 637bc92ba1 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Thu Apr 6 10:37:58 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514 commit 4b321aa5374427a15346be9b87b7c545ccb7db61 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 74105291cc687fe41b75de692fc624ab1335341a Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit 3df34605e078beb9babcbeadc7b119a70da63796 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation

commit ce540f943b8dae0786a87e51cafacb66efd1e64d Merge: 9815d6db04 fd5b300b57 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:45:27 2023 -0600 Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized commit 9815d6db04c264456b7ead7fc5f735513a868c7d Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:23:00 2023 -0600 Abstract Partition Assignor commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Jun 29 16:36:53 2023 -0700 Refactored code to reduce format conversion time commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 31 16:21:01 2023 -0700 test changes and code changes commit e1a032579129a2cd3aae1e93f46570bd57d3343c Author: Ritika Reddy <rreddy@confluent.io> Date: Tue May 16 09:28:57 2023 -0700 small changes commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu May 11 20:36:59 2023 -0700 Squashed commit of the following: commit c757af5f7c630d532bfee5f6dc45aec603ad8a29 Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri May 12 04:39:12 2023 +0200 KAFKA-14752: Kafka examples improvements - demo changes (#13517) KAFKA-14752: Kafka examples improvements - demo changes Reviewers: Luke Chen <showuon@gmail.com> commit 54a4067f81e1434d956ef797274f7b437fe49ea1 Author: Kamal Chandraprakash <kchandraprakash@uber.com> Date: Thu May 11 21:49:21 2023 +0530 KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060) Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org> commit bd65db82b4bad623b0bb31398979e466978148da Author: Josep Prat <josep.prat@aiven.io> Date: Thu May 11 17:55:26 2023 +0200 MINOR: clean up unused methods in core utils (#13706) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org> commit 38adb3956979d7025d148612975cc0b82200b2e1 Author: Jeff Kim <kimkb2011@gmail.com> Date: Thu May 11 07:51:41 2023 -0400 MINOR: add test tag for testDeadToDeadIllegalTransition (#13694) Reviewers: David Jacot <djacot@confluent.io> commit ee4132863553fb4fd2df715b2fbd77f349f978b8 Author: Federico Valeri <fedevaleri@gmail.com> Date: Thu May 11 11:19:32 2023 +0200 KAFKA-14752: Kafka examples improvements - processor changes (#13516) Reviewers: Luke Chen <showuon@gmail.com> commit a263627adb75f1ca5c87f1482cc70b994ba49d63 Author: Mickael Maison <mimaison@users.noreply.github.com> Date: Thu May 11 11:02:45 2023 +0200 MINOR: Remove unused methods in CoreUtils (#13170) Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com> commit 920a3601ffa7266edf12f3559cfe97c8a5929d03 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Thu May 11 04:13:29 2023 +0200 MINOR: fix a small typo in SharedServer.scala (#13693) Diabled -> Disabled commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc Author: A. Sophie Blee-Goldman <sophie@confluent.io> Date: Wed May 10 13:39:15 2023 -0700 HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695) The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen() Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io> commit f17fb75b2de32512f14cb94a7d1bfb0f37485780 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Wed May 10 16:41:52 2023 +0200 KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690) Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 4653507926a42dccda5c086fcae6278afcfc53ca Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Wed May 10 05:09:12 2023 -0700 KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443) This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers. Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io> commit 625ef176ee5f167786003c2d88498632b0b7014b Author: Luke Chen <showuon@gmail.com> Date: Wed May 10 16:40:20 2023 +0800 MINOR: remove kraft readme link (#13691) The config/kraft/README.md is already removed. We should also remove the link. Reviewers: dengziming <dengziming1993@gmail.com> commit 228434d23583189cdcaa7f4a90ebb178ccc17c73 Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue May 9 10:49:27 2023 -0400 KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644) This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java. Reviewers: David Jacot <djacot@confluent.io> commit 59ba9dbbc927ddc8660d0d98d9422909fd306758 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 9 17:58:45 2023 +0530 KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688) `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods. This refactoring changes no functionality other than restoring the older methods. Reviewers: Randall Hauch <rhauch@gmail.com> commit b40a7fc037bb1543c3355fad9c71570f770f5177 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon May 8 14:24:11 2023 -0700 HOTFIX: fix broken Streams upgrade system test (#13654) Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io> commit 7634eee2627da39937e3112ffc58bd7cfedc98f2 Author: David Jacot <djacot@confluent.io> Date: Mon May 8 20:46:07 2023 +0200 KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638) This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state. Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 86daf8ce6573eb79d6e78381dbab738055f914c4 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Mon May 8 20:09:47 2023 +0530 KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594) #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR. Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places. --------- Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6 Author: Christo Lolov <lolovc@amazon.com> Date: Mon May 8 15:24:52 2023 +0100 KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621) Migrates ChangeLogReader mock in TaskManagerTest to mockito. Reviewer: Bruno Cadonna <cadonna@apache.org> commit 347238948b86882a47faee4a2916d1b01333d95f Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Mon May 8 07:36:36 2023 +0100 KAFKA-14662: Update the ACL list in the doc (#13660) Added the missing ACLs to the doc. Reviewers: Luke Chen <showuon@gmail.com> commit a556def5724efb1dc96bd2d389411a8d2f802f53 Author: Divij Vaidya <diviv@amazon.com> Date: Mon May 8 08:35:14 2023 +0200 MINOR: Print the cause of failure for test (#13672) Motivation PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build): org.opentest4j.AssertionFailedError: expected: <false> but was: <true> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31) at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228) at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is. Change 1. Make the test a bit more robust by waiting for server startup. 2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure. Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com> commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 08:33:31 2023 +0200 MINOR: Fix producer Callback comment (#13669) Fixes the wrong exception name: OffsetMetadataTooLargeException. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com> commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399 Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 04:15:52 2023 +0200 KAFKA-14752: Kafka examples improvements - producer changes (#13515) KAFKA-14752: Kafka examples improvements - producer changes Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com> commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714 Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Sat May 6 02:56:26 2023 +0800 MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659) Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro> commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 14:05:20 2023 +0200 KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312) Motivation Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit. In this PR, we only update varInt32 encoding/decoding. Changes This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation. Results Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com> commit e34f88403159cc8381da23dafdf7e3d7403114a2 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 13:55:17 2023 +0200 KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623) When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io> commit 0822ce0ed1a106a510930bc9ac53a266f54684d7 Author: David Arthur <mumrah@gmail.com> Date: Fri May 5 04:35:26 2023 -0400 KAFKA-14840: Support for snapshots during ZK migration (#13461) This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active controller loaded a snapshot, it would get out of sync with the ZK state. In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass and without using too much memory. Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com> commit 97c36f3f3142580325daa1a6aadb662893390561 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 12:20:33 2023 -0700 HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540 commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 11:18:03 2023 -0700 MINOR: improve QuorumController logging #13540 When creating the QuorumController, log whether ZK migration is enabled. When applying a feature level record which sets the metadata version, log the metadata version enum rather than the numeric feature level. Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the end of replay. When TRACE is enabled, log every record that is replayed in QuorumController. Since some records may contain sensitive information, create RecordRedactor to assist in logging only what is safe to put in the log4j file. Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged at INFO, and additions are logged at TRACE. Remove SnapshotReason.java, SnapshotReasonTest.java, and QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to org.apache.kafka.image.publisher.SnapshotGenerator. Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff Author: Justine Olshan <jolshan@confluent.io> Date: Thu May 4 09:55:45 2023 -0700 KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607) Also modifies verification to only add a partition to verify if it is transactional. When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different. Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit ea81e99e5980c807414651034a8c60426a158ca4 Author: Philip Nee <pnee@confluent.io> Date: Thu May 4 09:20:01 2023 -0700 KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149) Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer. This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin. Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest. Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dc7819d7f1fe6b0160cd95246420ab10c335410b Author: Christo Lolov <lolovc@amazon.com> Date: Thu May 4 11:00:33 2023 +0100 KAFKA-14594: Move LogDirsCommand to tools module (#13122) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d46c3f259cce25c43f20fba3943d5cb34ed909ea Author: David Mao <47232755+splett2@users.noreply.github.com> Date: Wed May 3 17:09:43 2023 -0700 MINOR: Reduce number of threads created for integration test brokers (#13655) The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker. Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io> commit c08120f83f7318f15dcf14d525876d18caf6afd0 Author: Jason Gustafson <jason@confluent.io> Date: Wed May 3 15:25:32 2023 -0700 MINOR: Allow tagged fields with version subset of flexible version range (#13551) The generated message types are missing a range check for the case when the tagged version range is a subset of the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the number of tags serialized. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b620c03ccf48d6d92b219cba35bb1e5e248d2547 Author: Luke Chen <showuon@gmail.com> Date: Thu May 4 01:08:25 2023 +0800 KAFKA-14946: fix NPE when merging the deltatable (#13653) Fix NPE while merging the deltatable. Because it's possible that hashTier is not null but deltatable is null (ex: removing data), we should have null check while merging for deltatable like other places did. Also added tests that will fail without this change. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 4a0b6ebf60ed7614f042443460b490971e8662a4 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 2 23:16:46 2023 +0530 KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657) Reviewers: Chris Egerton <chrise@aiven.io> commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e Author: David Jacot <djacot@confluent.io> Date: Tue May 2 18:04:50 2023 +0200 KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637) This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group. Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949 Author: Christo Lolov <lolovc@amazon.com> Date: Tue May 2 16:39:31 2023 +0100 MINOR: Remove unnecessary code in client/connect (#13259) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3 Author: Federico Valeri <fedevaleri@gmail.com> Date: Tue May 2 14:28:42 2023 +0200 KAFKA-14752: Kafka examples improvements - consumer changes (#13514) KAFKA-14752: Kafka examples improvements - consumer changes This is extracted from the original PR for better review. https://github.com/apache/kafka/pull/13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com> Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com> commit 141c76a2c904705f2cd484e96767fcb217c5db25 Author: Bruno Cadonna <cadonna@apache.org> Date: Tue May 2 14:00:34 2023 +0200 KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529) 1. Migrates topology builder mock in TaskManagerTest to mockito. 2. Replaces the unit test to verify if subscribed partitions are added to topology metadata. 3. Modifies signatures of methods for adding subscribed partitions to topology metadata to use sets instead of lists. This makes the intent of the methods clearer and makes the tests more portable. Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org> commit 21af1918eafa30812a955c3c0295b9e968841cd3 Author: Luke Chen <showuon@gmail.com> Date: Tue May 2 09:54:12 2023 +0800 MINOR: Add reason to exceptions in QuorumController (#13648) Saw this error message in log: ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController) The null reason is not helpful with only RejectedExecutionException. Adding the reason to it. Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit 4773961a44d1f0d1e11d662c7e0fc955027bced2 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon May 1 21:51:09 2023 +0530 MINOR: Fix Javadoc for configureAdminResources in Connect's RestServer (#13635) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io> commit e29942347acc70aa85d47e84e2021f9c24cd7c80 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Mon May 1 09:56:04 2023 -0400 KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628) Handle migrating SCRAM records in ZK when migrating from ZK to KRaft. This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes. Reviewers: David Arthur <mumrah@gmail.com> commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 08:46:40 2023 -0700 MINOR: Fixing typos in the ConsumerCoordinator (#13618) Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io> commit e55fbceb6667cc1455f7fb2e96421c85741fa7df Author: Anton Agestam <anton.agestam@aiven.io> Date: Fri Apr 28 11:54:28 2023 +0100 MINOR: Fix incorrect description of SessionLifetimeMs (#13649) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984 Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 02:08:32 2023 -0700 KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550) This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS. This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method. In short, it should no long check and validate only the highest generation. Instead, we consider 3 cases: 1. Member will continue to hold on to its partition if there are no other owners 2. If there are 1+ owners to the same partition. One with the highest generation will win. 3. If two members of the same generation hold on to the same partition. We will log an error but remove both from the assignment. (Same with the current logic) Here are some important notes that lead to the patch: - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown. - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error. This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive To summarize the current behavior of different errors: `REBALANCE_IN_PROGRESS` - heartbeat: requestRejoin if member state is stable - joinGroup: rejoin immediately - syncGroup: rejoin immediately - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress. `UNKNOWN_MEMBER_ID` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately - syncGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately `ILLEGAL_GENERATION` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing. then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately Reviewers: David Jacot <djacot@confluent.io> commit 10b3e667132934084a2d275a204a1a782c2df94e Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri Apr 28 10:32:11 2023 +0200 KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d796480fe87fd819fc0ac560ca318759180d4644 Author: Luke Chen <showuon@gmail.com> Date: Fri Apr 28 14:35:12 2023 +0800 KAFKA-14909: check zkMigrationReady tag before migration (#13631) 1. add ZkMigrationReady in apiVersionsResponse 2. check all nodes if ZkMigrationReady are ready before moving to next migration state Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com> commit c708f7ba5f4f449920cec57a5b69e84e92128b54 Author: Colin Patrick McCabe <cmccabe@apache.org> Date: Thu Apr 27 19:15:26 2023 -0700 MINOR: remove spurious call to fatalFaultHandler (#13651) Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805. We should only invoke the fatal fault handller if we are unable to generate the activation records. If we are unable to write the activation records, a controller failover should be sufficient to remedy the situation. Co-authored-by: Luke Chen showuon@gmail.com Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com> commit 056657d84d84e116ffc9460872945b4d2b479ff3 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 17:04:56 2023 -0400 MINOR add license to reviewers.py commit a08f31ecfefca1a51d64137a344e62a236740e62 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 14:32:19 2023 -0400 MINOR: Adding reviewers.py (#11096) This script can be used to help build the "Reviewers: " string we include in commit messages. Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit 70493336171363cfa95237a0fe14ef57090553e4 Author: Colin P. McCabe <cmccabe@apache.org> Date: Wed Apr 26 16:10:46 2023 -0700 KAFKA-14943: Fix ClientQuotaControlManager validation Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of internal ZK representation details that treated them both as "client configs.") Add unit tests for ClientQuotaControlManager.isValidIpEntity and ClientQuotaControlManager.configKeysForEntityType. This change doesn't affect metadata record application, only input validation. If there are bad client quotas that are set currently, this change will not alter the current behavior (of throwing an exception and ignoring the bad quota). commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051 Author: David Jacot <djacot@confluent.io> Date: Thu Apr 27 14:05:41 2023 +0200 KAFKA-14462; [9/N] Add RecordHelpers (#13544) This patch adds RecordHelpers. Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9 Author: LinShunKang <linshunkang.chn@gmail.com> Date: Thu Apr 27 10:44:08 2023 +0800 KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545) This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035 Direct use ByteBuffer instead of byte[] to deserialize. Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro> commit c1b5c75d9271638776392822a094e9e7ef37f490 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 26 10:20:30 2023 -0400 KAFKA-14805 KRaft controller supports pre-migration mode (#13407) This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode, the controller will only allow certain write operations. The purpose of this is to disallow metadata changes when the controller is waiting for the ZK migration records to be committed. The following ControllerWriteEvent operations are permitted in pre-migration mode * completeActivation * maybeFenceReplicas * writeNoOpRecord * processBrokerHeartbeat * registerBroker (only for migrating ZK brokers) * unregisterBroker Raft events and other controller events do not follow the same code path as ControllerWriteEvent, so they are not affected by this new behavior. This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean MIGRATION rather than PRE_MIGRATION. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 007c0d375a6e70aefb65f58f9f096016c4cbea16 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 26 17:53:56 2023 +0530 KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634) Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit baf127a6633161cb52747467880b006d2f54d3bd Author: Greg Harris <greg.harris@aiven.io> Date: Wed Apr 26 00:30:13 2023 -0700 KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429) Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io> commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef Author: Victoria Xia <victoria.xia@confluent.io> Date: Tue Apr 25 22:39:23 2023 -0400 KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622) Reviewers: Matthias J. Sax <matthias@confluent.io> commit a7d0b3f753708a93aea92e614833f6f6e7443234 Author: Said Boudjelda <bmscomp@gmail.com> Date: Tue Apr 25 23:31:04 2023 +0200 MINOR: Upgrade gradle to 8.1.1 (#13625) Also upgrade gradle plugins: - `org.owasp.dependencycheck` gradle plugin to version `8.2.1` - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1` Gradle release notes: * https://docs.gradle.org/8.1.1/release-notes.html Reviewers: Ismael Juma <ismael@juma.me.uk> commit 9a36da12b7359b7158332c541655716312efb5b3 Author: David Jacot <djacot@confluent.io> Date: Tue Apr 25 18:50:51 2023 +0200 KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538) This patch adds ConsumerGroupMember. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io> commit 4780dc773f2cd5a20fa5be38d20137745690a888 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue Apr 25 21:20:35 2023 +0530 KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4 Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Tue Apr 25 13:28:37 2023 +0100 KAFKA-14592: Move FeatureCommand to tools (#13459) KAFKA-14592: Move FeatureCommand to tools Reviewers: Luke Chen <showuon@gmail.com> commit d83a734c41c43a03aa571e602cf8535f0893fd79 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 25 12:02:24 2023 +0200 MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626) The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call. Reviewers: maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com> commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 15:29:57 2023 -0700 KAFKA-12446: update change encoding to use varint (#13533) KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io> commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402 Author: Victoria Xia <victoria.xia@confluent.io> Date: Mon Apr 24 17:06:26 2023 -0400 KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615) Reviewers: Matthias J. Sax <matthias@confluent.io> commit 6dcdb017327587a6943fa868595fa3488c7f7ef7 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 12:40:25 2023 -0700 KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592) Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks. This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io> commit 2271e748a11919d07698ebce759dca2e3075596a Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Mon Apr 24 17:21:19 2023 +0800 MINOR: fix zookeeper_migration_test.py (#13620) Reviewers: Mickael Maison <mimaison@users.noreply.github.com> commit c3241236296f9aaee103e475f242e32f04d4c256 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon Apr 24 14:06:20 2023 +0530 KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 7061475445cb6314e7cf4f9848384224b14f4395 Author: Greg Harris <greg.harris@aiven.io> Date: Fri Apr 21 12:55:41 2023 -0700 KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575) Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest` Reviewers: Josep Prat <jlprat@apache.org> commit ecdef88f744410039b88e90a5979078d5735aa06 Author: Matthias J. Sax <matthias@confluent.io> Date: Fri Apr 21 12:48:05 2023 -0700 MINOR: updated KS release notes for 3.5 (#13577) Reviewers: Walker Carlson <wcarlson@confluent.io> commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Fri Apr 21 15:02:06 2023 +0200 MINOR: fix noticed typo in raft and metadata projects (#13612) Reviewers: Josep Prat <jlprat@apache.org> commit c39bf714bbaf2a632be4c2a7e446553fe40ba129 Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:22:16 2023 +0200 KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537) This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 2d0b816150c79057c813387bd126523b6326a1fc Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:19:04 2023 +0200 MINOR: Move `ControllerPurgatory` to `server-common` (#13555) This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit df137752542c005c6998c37c03222ffbeca0f349 Author: Purshotam Chauhan <pchauhan@confluent.io> Date: Fri Apr 21 14:08:23 2023 +0530 KAFKA-14828: Remove R/W locks using persistent data structures (#13437) Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only. In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation. Reviewers: Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Divij Vaidya <diviv@amazon.com> commit 2ee770ac7e576c85a911ccb307339be3d58a8942 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu Apr 20 10:21:26 2023 -0700 Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)" This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc. commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3 Author: Justine Olshan <jolshan@confluent.io> Date: Thu Apr 20 09:30:11 2023 -0700 KAFKA-14904: Pending state blocked verification of transactions (#13579) KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction. There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes. When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions. The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely. This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed. Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit 7f175feacaba4187bbb3631dff3a1330060f191e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Thu Apr 20 13:50:29 2023 +0530 KAFKA-14586: Adding redirection for StreamsResetter (#13614) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com> commit e14dd8024adae746c50c8b7d9cd268e859669576 Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com> Date: Thu Apr 20 05:29:27 2023 +0200 KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432) We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic: 1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry. 2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of the client. 3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`. We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them. Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io> commit f5de4daa71f0ad31aa64443c5ff43c59712feea5 Author: Ron Dagostino <rndgstn@gmail.com> Date: Wed Apr 19 19:45:26 2023 -0400 KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change events. However, it does not distinguish between issues related to feature changes actually failing and other exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues. We shutdown the broker only for FeatureCacheUpdateException. Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 11c8bf4826197533807b2132cfc6599ba70de1c1 Author: Victoria Xia <victoria.xia@confluent.io> Date: Wed Apr 19 19:34:36 2023 -0400 KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609) This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter. Part of KIP-914. Reviewer: Matthias J. Sax <matthias@confluent.io> commit 809966a9a06664e0b521c2298fa0de834e443607 Author: Matthew de Detrich <matthew.dedetrich@aiven.io> Date: Wed Apr 19 20:54:07 2023 +0200 KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478) Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address. Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org> commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 19 14:19:13 2023 -0400 KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606) This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b10716e72370c4e128bddb17bcc107ccab221e47 Author: hudeqi <1217150961@qq.com> Date: Thu Apr 20 00:49:08 2023 +0800 KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com> commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc Author: Keith Wall <kwall@redhat.com> Date: Wed Apr 19 03:58:29 2023 +0100 KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572) Changes SocketServer to set the setReuseAddress(true) socket option. This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test. Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com> commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 19 00:46:03 2023 +0530 MINOR: Fixing gradle build during compileScala and compileTestScala (#13588) Reviewers: Chia-Ping Tsai <chia7712@gmail.com> commit 3388adf1b52f545e2d69edd30cdd53241b2887f3 Author: Matthias J. Sax <matthias@confluent.io> Date: Tue Apr 18 11:32:27 2023 -0700 MINOR: rename internal FK-join processor classes (#13589) Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io> commit abca86511ecc9e081d676976ff6d3b845308f444 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Tue Apr 18 12:41:38 2023 -0400 KAFKA-14881: Rework UserScramCredentialRecord (#13513) Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This is necessary to support migration from ZK, since those are the fields we stored in ZK. Update latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version. Moved ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more appropriate. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 61530d68ce83467de6190a52da37b3c0af84f0ef Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 09:37:04 2023 -0400 KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526) This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields. Reviewers: David Jacot <djacot@confluent.io> commit b36a170aa3b2738177d7229859db765e0115385b Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 18 13:36:56 2023 +0200 MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593) Reviewers: Luke Chen <showuon@gmail.com> commit 5e9d4de748dff7b91043a9b799716ab4becdae7e Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 04:41:54 2023 -0400 KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511) This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io> commit 454b72161a76b1687a1263157d7cc30a1bdb2506 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Tue Apr 18 09:40:14 2023 +0200 KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557) KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow. To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread. commit e27926f92b1f6b34ed6731f33c712a5d0d594275 Author: Ron Dagostino <rndgstn@gmail.com> Date: Mon Apr 17 17:52:28 2023 -0400 KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io> commit 7159f6c1a81ead277030f55f293943270346ad4e Author: Alyssa Huang <ahuang@confluent.io> Date: Mon Apr 17 11:03:45 2023 -0700 MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it should have been. Add a regression test. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit cc88822986e4004e772f376eae47dbe08e29d137 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 10 16:29:27 2023 -0700 small changes commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 25 14:58:20 2023 -0700 grammar and code comments changes commit 5550e4934ade28f069e4ae9ab97e76982ff3401f Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 17:39:48 2023 -0700 fixed formatting stuff commit 28ab29819bf327309ce07196a3fb0492f7fb2281 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 11:38:57 2023 -0700 consumers -> members commit e56ded267456bbe1c22ded0d037953545b921d1d Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 11:26:23 2023 -0700 Interface changes incorporated commit 2d50c278c18baa11b0235330149dd8f9def57c3d Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 13 14:47:33 2023 -0700 Renamed topicIdToPartition class commit d33bbfac053ccac77204ac4253698ed356d7c621 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 17:31:26 2023 -0700 Added stickiness test commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 13:28:24 2023 -0700 Separated uniform and general builder in different files, added more tests commit ef814867aba68fc4813359c5cef6ae155ed49a60 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:24:27 2023 -0700 import changes commit dffad677c6e4a33625401d621f046af6f21feb99 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:16:33 2023 -0700 Fixed checkstyle and added test file commit 7fd779418df3504a490c0d10f3848360a4417077 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 5 15:13:38 2023 -0700 First draft optimised uniform assignor code commit 309cc6141f7962e794c90d4107f460fafd51697d Merge: 3d9264547c 2c1cf03a89 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Mon Apr 17 11:03:07 2023 -0700 Merge branch 'trunk' into rreddy-22/KAFKA-14514 commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 10:58:16 2023 -0700 Interface changes incorporated, added stickiness test and non-existent topic test. commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436 Merge: a1bdb58e08 57b94ca208 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 09:49:23 2023 -0700 Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514 # Conflicts: # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java commit a1bdb58e082d98b73f50860db5b35684f4eba51b Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 20:01:59 2023 -0700 Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc. commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 11:28:40 2023 -0700 Addressed some PR comments commit 918d262ae52892c13766649db847ede6928921f8 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7369db3744b4c89f625f6b73db331eaf0512c4eb Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit a925b02dd3d78c8b9c1415c083f0544edcb404fe Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit 48a982232c6f1ff371cb44333fd12d6b37ef381a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 13:00:07 2023 -0700 Removed topicIdToPartition class, changed attribute names and added java doc for getter methods commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 15:09:44 2023 -0700 Made all attributes private and added getter methods, changed names according to PR comments commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:25:33 2023 -0700 Added toString method and hash commit 5065109332845541c5591a52683f21d37982d80a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 13:43:26 2023 -0700 Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure commit 57b94ca208cb374b22d61eb044ef8004ea09479f Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7352cbb5683f676382f56e6ae5c926f3cae98824 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0 Merge: 4b321aa537 637bc92ba1 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Thu Apr 6 10:37:58 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514 commit 4b321aa5374427a15346be9b87b7c545ccb7db61 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 74105291cc687fe41b75de692fc624ab1335341a Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit 3df34605e078beb9babcbeadc7b119a70da63796 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation

commit 71f8488e5bf30af6f4a465d1fac52ccb9a341396 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 25 15:22:10 2023 -0700 Deleted PartitionMetadata and added TopicMetadata to the SubscribedTopicMetadata, PR comments commit 5a935324c11e6f6309ee5451f90ed5052276a734 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 25 10:51:11 2023 -0700 Minor commit 1c955b5e251aedc174cf28f43037f08162d89b49 Merge: 2d40125fa8 81f1ccd7a2 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 13:02:54 2023 -0700 Merge branch 'rreddy-22/Rack-Assignor-Interface-Changes' of github.com:rreddy-22/kafka-rreddy into rreddy-22/Rack-Assignor-Interface-Changes commit 81f1ccd7a25d85eaf1cff713df402e0100d4b4a3 Merge: a39e76f63f 84691b11f6 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Mon Jul 24 12:56:40 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes commit 2d40125fa8ba16b4f07e6bfe410ceacfd8a112e8 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 11:45:21 2023 -0700 minor commit a39e76f63f312ecacdcebd2c5015777e160ca0b3 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 10:04:10 2023 -0700 reverted reviewers.py changes commit c0b464c8936f1df4f6540b8abe7c1fa88fce6e81 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Jul 24 00:02:03 2023 -0700 reverted grammar changes commit 6fe72dfce921ab9d77ba6b28d4104c5d92c6dbf3 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 23:56:35 2023 -0700 minor commit a30a53463e55b72699617360d1c888b7ee310336 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 23:55:35 2023 -0700 minor commit 57b9a6233a4f697f285b85e6394d3dd407f16404 Merge: 061dac797b 4981fa939d Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Sun Jul 23 19:52:07 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes commit 061dac797bae56d91659fbc8d7900117eb10a866 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:43:12 2023 -0700 Moved SubscribedTopicMetadata to the consumer package commit 6650ea5cd707f03713fc2c69ba76fafd56d886b8 Merge: 22f2df02de faafed25a1 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:33:17 2023 -0700 Merge remote-tracking branch 'origin/rreddy-22/Rack-Assignor-Interface-Changes' into rreddy-22/Rack-Assignor-Interface-Changes commit 22f2df02deabe1a6b5b8b8cceb432486b9ef4470 Author: Ritika Reddy <rreddy@confluent.io> Date: Sun Jul 23 19:25:11 2023 -0700 minor changes commit faafed25a10d7ff79fb8a24ab62aa222c0e6516a Merge: 4511683d4e 1bf73d89d0 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Fri Jul 21 12:50:59 2023 -0700 Merge branch 'trunk' into rreddy-22/Rack-Assignor-Interface-Changes commit 4511683d4ef489cb979769ff6775dfe0250cdeeb Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 21 03:09:27 2023 -0700 Changes based on PR comments commit 579bb948fcc78afd163410a5bc43832bc87aaf71 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Jul 19 14:45:18 2023 -0700 New interfaces and classes added to facilitate passing rack information to the assignor, modified tests to incorporate changes commit f68471a99f2e0260b36047a7596b0f9de48794ad Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Jul 13 11:55:20 2023 -0700 Small edits commit c50f3bdbf79f1ca127355d2a9002b1fdbf0baa2f Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 11 16:30:32 2023 -0700 Removed extra lines commit 478488e0943aa879b27eec5ae5eb79a23dd9cab3 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Jul 11 16:21:54 2023 -0700 Added rack interface changes, added a new TopicAndClusterMetadata class to handle topic and cluster images and added an AbstractPartitionAssignor commit ce540f943b8dae0786a87e51cafacb66efd1e64d Merge: 9815d6db04 fd5b300b57 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:45:27 2023 -0600 Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized commit 9815d6db04c264456b7ead7fc5f735513a868c7d Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Jul 7 12:23:00 2023 -0600 Abstract Partition Assignor commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Jun 29 16:36:53 2023 -0700 Refactored code to reduce format conversion time commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 31 16:21:01 2023 -0700 test changes and code changes commit e1a032579129a2cd3aae1e93f46570bd57d3343c Author: Ritika Reddy <rreddy@confluent.io> Date: Tue May 16 09:28:57 2023 -0700 small changes commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu May 11 20:36:59 2023 -0700 Squashed commit of the following: commit c757af5f7c630d532bfee5f6dc45aec603ad8a29 Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri May 12 04:39:12 2023 +0200 KAFKA-14752: Kafka examples improvements - demo changes (#13517) KAFKA-14752: Kafka examples improvements - demo changes Reviewers: Luke Chen <showuon@gmail.com> commit 54a4067f81e1434d956ef797274f7b437fe49ea1 Author: Kamal Chandraprakash <kchandraprakash@uber.com> Date: Thu May 11 21:49:21 2023 +0530 KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060) Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org> commit bd65db82b4bad623b0bb31398979e466978148da Author: Josep Prat <josep.prat@aiven.io> Date: Thu May 11 17:55:26 2023 +0200 MINOR: clean up unused methods in core utils (#13706) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org> commit 38adb3956979d7025d148612975cc0b82200b2e1 Author: Jeff Kim <kimkb2011@gmail.com> Date: Thu May 11 07:51:41 2023 -0400 MINOR: add test tag for testDeadToDeadIllegalTransition (#13694) Reviewers: David Jacot <djacot@confluent.io> commit ee4132863553fb4fd2df715b2fbd77f349f978b8 Author: Federico Valeri <fedevaleri@gmail.com> Date: Thu May 11 11:19:32 2023 +0200 KAFKA-14752: Kafka examples improvements - processor changes (#13516) Reviewers: Luke Chen <showuon@gmail.com> commit a263627adb75f1ca5c87f1482cc70b994ba49d63 Author: Mickael Maison <mimaison@users.noreply.github.com> Date: Thu May 11 11:02:45 2023 +0200 MINOR: Remove unused methods in CoreUtils (#13170) Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com> commit 920a3601ffa7266edf12f3559cfe97c8a5929d03 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Thu May 11 04:13:29 2023 +0200 MINOR: fix a small typo in SharedServer.scala (#13693) Diabled -> Disabled commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc Author: A. Sophie Blee-Goldman <sophie@confluent.io> Date: Wed May 10 13:39:15 2023 -0700 HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695) The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen() Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io> commit f17fb75b2de32512f14cb94a7d1bfb0f37485780 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Wed May 10 16:41:52 2023 +0200 KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690) Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 4653507926a42dccda5c086fcae6278afcfc53ca Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Wed May 10 05:09:12 2023 -0700 KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443) This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers. Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io> commit 625ef176ee5f167786003c2d88498632b0b7014b Author: Luke Chen <showuon@gmail.com> Date: Wed May 10 16:40:20 2023 +0800 MINOR: remove kraft readme link (#13691) The config/kraft/README.md is already removed. We should also remove the link. Reviewers: dengziming <dengziming1993@gmail.com> commit 228434d23583189cdcaa7f4a90ebb178ccc17c73 Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue May 9 10:49:27 2023 -0400 KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644) This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java. Reviewers: David Jacot <djacot@confluent.io> commit 59ba9dbbc927ddc8660d0d98d9422909fd306758 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 9 17:58:45 2023 +0530 KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688) `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods. This refactoring changes no functionality other than restoring the older methods. Reviewers: Randall Hauch <rhauch@gmail.com> commit b40a7fc037bb1543c3355fad9c71570f770f5177 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon May 8 14:24:11 2023 -0700 HOTFIX: fix broken Streams upgrade system test (#13654) Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io> commit 7634eee2627da39937e3112ffc58bd7cfedc98f2 Author: David Jacot <djacot@confluent.io> Date: Mon May 8 20:46:07 2023 +0200 KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638) This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state. Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 86daf8ce6573eb79d6e78381dbab738055f914c4 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Mon May 8 20:09:47 2023 +0530 KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594) #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR. Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places. --------- Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6 Author: Christo Lolov <lolovc@amazon.com> Date: Mon May 8 15:24:52 2023 +0100 KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621) Migrates ChangeLogReader mock in TaskManagerTest to mockito. Reviewer: Bruno Cadonna <cadonna@apache.org> commit 347238948b86882a47faee4a2916d1b01333d95f Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Mon May 8 07:36:36 2023 +0100 KAFKA-14662: Update the ACL list in the doc (#13660) Added the missing ACLs to the doc. Reviewers: Luke Chen <showuon@gmail.com> commit a556def5724efb1dc96bd2d389411a8d2f802f53 Author: Divij Vaidya <diviv@amazon.com> Date: Mon May 8 08:35:14 2023 +0200 MINOR: Print the cause of failure for test (#13672) Motivation PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build): org.opentest4j.AssertionFailedError: expected: <false> but was: <true> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36) at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31) at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228) at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is. Change 1. Make the test a bit more robust by waiting for server startup. 2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure. Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com> commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 08:33:31 2023 +0200 MINOR: Fix producer Callback comment (#13669) Fixes the wrong exception name: OffsetMetadataTooLargeException. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com> commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399 Author: Federico Valeri <fedevaleri@gmail.com> Date: Mon May 8 04:15:52 2023 +0200 KAFKA-14752: Kafka examples improvements - producer changes (#13515) KAFKA-14752: Kafka examples improvements - producer changes Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com> commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714 Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Sat May 6 02:56:26 2023 +0800 MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659) Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro> commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 14:05:20 2023 +0200 KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312) Motivation Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit. In this PR, we only update varInt32 encoding/decoding. Changes This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation. Results Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com> commit e34f88403159cc8381da23dafdf7e3d7403114a2 Author: Divij Vaidya <diviv@amazon.com> Date: Fri May 5 13:55:17 2023 +0200 KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623) When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective. Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io> commit 0822ce0ed1a106a510930bc9ac53a266f54684d7 Author: David Arthur <mumrah@gmail.com> Date: Fri May 5 04:35:26 2023 -0400 KAFKA-14840: Support for snapshots during ZK migration (#13461) This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active controller loaded a snapshot, it would get out of sync with the ZK state. In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass and without using too much memory. Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com> commit 97c36f3f3142580325daa1a6aadb662893390561 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 12:20:33 2023 -0700 HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540 commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu May 4 11:18:03 2023 -0700 MINOR: improve QuorumController logging #13540 When creating the QuorumController, log whether ZK migration is enabled. When applying a feature level record which sets the metadata version, log the metadata version enum rather than the numeric feature level. Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the end of replay. When TRACE is enabled, log every record that is replayed in QuorumController. Since some records may contain sensitive information, create RecordRedactor to assist in logging only what is safe to put in the log4j file. Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged at INFO, and additions are logged at TRACE. Remove SnapshotReason.java, SnapshotReasonTest.java, and QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to org.apache.kafka.image.publisher.SnapshotGenerator. Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff Author: Justine Olshan <jolshan@confluent.io> Date: Thu May 4 09:55:45 2023 -0700 KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607) Also modifies verification to only add a partition to verify if it is transactional. When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different. Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit ea81e99e5980c807414651034a8c60426a158ca4 Author: Philip Nee <pnee@confluent.io> Date: Thu May 4 09:20:01 2023 -0700 KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149) Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer. This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin. Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest. Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dc7819d7f1fe6b0160cd95246420ab10c335410b Author: Christo Lolov <lolovc@amazon.com> Date: Thu May 4 11:00:33 2023 +0100 KAFKA-14594: Move LogDirsCommand to tools module (#13122) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d46c3f259cce25c43f20fba3943d5cb34ed909ea Author: David Mao <47232755+splett2@users.noreply.github.com> Date: Wed May 3 17:09:43 2023 -0700 MINOR: Reduce number of threads created for integration test brokers (#13655) The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker. Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io> commit c08120f83f7318f15dcf14d525876d18caf6afd0 Author: Jason Gustafson <jason@confluent.io> Date: Wed May 3 15:25:32 2023 -0700 MINOR: Allow tagged fields with version subset of flexible version range (#13551) The generated message types are missing a range check for the case when the tagged version range is a subset of the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the number of tags serialized. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b620c03ccf48d6d92b219cba35bb1e5e248d2547 Author: Luke Chen <showuon@gmail.com> Date: Thu May 4 01:08:25 2023 +0800 KAFKA-14946: fix NPE when merging the deltatable (#13653) Fix NPE while merging the deltatable. Because it's possible that hashTier is not null but deltatable is null (ex: removing data), we should have null check while merging for deltatable like other places did. Also added tests that will fail without this change. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 4a0b6ebf60ed7614f042443460b490971e8662a4 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue May 2 23:16:46 2023 +0530 KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657) Reviewers: Chris Egerton <chrise@aiven.io> commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e Author: David Jacot <djacot@confluent.io> Date: Tue May 2 18:04:50 2023 +0200 KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637) This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group. Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949 Author: Christo Lolov <lolovc@amazon.com> Date: Tue May 2 16:39:31 2023 +0100 MINOR: Remove unnecessary code in client/connect (#13259) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3 Author: Federico Valeri <fedevaleri@gmail.com> Date: Tue May 2 14:28:42 2023 +0200 KAFKA-14752: Kafka examples improvements - consumer changes (#13514) KAFKA-14752: Kafka examples improvements - consumer changes This is extracted from the original PR for better review. https://github.com/apache/kafka/pull/13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com> Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com> commit 141c76a2c904705f2cd484e96767fcb217c5db25 Author: Bruno Cadonna <cadonna@apache.org> Date: Tue May 2 14:00:34 2023 +0200 KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529) 1. Migrates topology builder mock in TaskManagerTest to mockito. 2. Replaces the unit test to verify if subscribed partitions are added to topology metadata. 3. Modifies signatures of methods for adding subscribed partitions to topology metadata to use sets instead of lists. This makes the intent of the methods clearer and makes the tests more portable. Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org> commit 21af1918eafa30812a955c3c0295b9e968841cd3 Author: Luke Chen <showuon@gmail.com> Date: Tue May 2 09:54:12 2023 +0800 MINOR: Add reason to exceptions in QuorumController (#13648) Saw this error message in log: ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController) The null reason is not helpful with only RejectedExecutionException. Adding the reason to it. Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit 4773961a44d1f0d1e11d662c7e0fc955027bced2 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon May 1 21:51:09 2023 +0530 MINOR: Fix Javadoc for configureAdminResources in Connect's RestServer (#13635) Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io> commit e29942347acc70aa85d47e84e2021f9c24cd7c80 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Mon May 1 09:56:04 2023 -0400 KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628) Handle migrating SCRAM records in ZK when migrating from ZK to KRaft. This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes. Reviewers: David Arthur <mumrah@gmail.com> commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 08:46:40 2023 -0700 MINOR: Fixing typos in the ConsumerCoordinator (#13618) Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io> commit e55fbceb6667cc1455f7fb2e96421c85741fa7df Author: Anton Agestam <anton.agestam@aiven.io> Date: Fri Apr 28 11:54:28 2023 +0100 MINOR: Fix incorrect description of SessionLifetimeMs (#13649) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984 Author: Philip Nee <pnee@confluent.io> Date: Fri Apr 28 02:08:32 2023 -0700 KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550) This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS. This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method. In short, it should no long check and validate only the highest generation. Instead, we consider 3 cases: 1. Member will continue to hold on to its partition if there are no other owners 2. If there are 1+ owners to the same partition. One with the highest generation will win. 3. If two members of the same generation hold on to the same partition. We will log an error but remove both from the assignment. (Same with the current logic) Here are some important notes that lead to the patch: - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown. - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error. This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive To summarize the current behavior of different errors: `REBALANCE_IN_PROGRESS` - heartbeat: requestRejoin if member state is stable - joinGroup: rejoin immediately - syncGroup: rejoin immediately - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress. `UNKNOWN_MEMBER_ID` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately - syncGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately `ILLEGAL_GENERATION` - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing. then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately Reviewers: David Jacot <djacot@confluent.io> commit 10b3e667132934084a2d275a204a1a782c2df94e Author: Federico Valeri <fedevaleri@gmail.com> Date: Fri Apr 28 10:32:11 2023 +0200 KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit d796480fe87fd819fc0ac560ca318759180d4644 Author: Luke Chen <showuon@gmail.com> Date: Fri Apr 28 14:35:12 2023 +0800 KAFKA-14909: check zkMigrationReady tag before migration (#13631) 1. add ZkMigrationReady in apiVersionsResponse 2. check all nodes if ZkMigrationReady are ready before moving to next migration state Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com> commit c708f7ba5f4f449920cec57a5b69e84e92128b54 Author: Colin Patrick McCabe <cmccabe@apache.org> Date: Thu Apr 27 19:15:26 2023 -0700 MINOR: remove spurious call to fatalFaultHandler (#13651) Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805. We should only invoke the fatal fault handller if we are unable to generate the activation records. If we are unable to write the activation records, a controller failover should be sufficient to remedy the situation. Co-authored-by: Luke Chen showuon@gmail.com Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com> commit 056657d84d84e116ffc9460872945b4d2b479ff3 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 17:04:56 2023 -0400 MINOR add license to reviewers.py commit a08f31ecfefca1a51d64137a344e62a236740e62 Author: David Arthur <mumrah@gmail.com> Date: Thu Apr 27 14:32:19 2023 -0400 MINOR: Adding reviewers.py (#11096) This script can be used to help build the "Reviewers: " string we include in commit messages. Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit 70493336171363cfa95237a0fe14ef57090553e4 Author: Colin P. McCabe <cmccabe@apache.org> Date: Wed Apr 26 16:10:46 2023 -0700 KAFKA-14943: Fix ClientQuotaControlManager validation Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of internal ZK representation details that treated them both as "client configs.") Add unit tests for ClientQuotaControlManager.isValidIpEntity and ClientQuotaControlManager.configKeysForEntityType. This change doesn't affect metadata record application, only input validation. If there are bad client quotas that are set currently, this change will not alter the current behavior (of throwing an exception and ignoring the bad quota). commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051 Author: David Jacot <djacot@confluent.io> Date: Thu Apr 27 14:05:41 2023 +0200 KAFKA-14462; [9/N] Add RecordHelpers (#13544) This patch adds RecordHelpers. Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9 Author: LinShunKang <linshunkang.chn@gmail.com> Date: Thu Apr 27 10:44:08 2023 +0800 KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545) This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035 Direct use ByteBuffer instead of byte[] to deserialize. Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro> commit c1b5c75d9271638776392822a094e9e7ef37f490 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 26 10:20:30 2023 -0400 KAFKA-14805 KRaft controller supports pre-migration mode (#13407) This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode, the controller will only allow certain write operations. The purpose of this is to disallow metadata changes when the controller is waiting for the ZK migration records to be committed. The following ControllerWriteEvent operations are permitted in pre-migration mode * completeActivation * maybeFenceReplicas * writeNoOpRecord * processBrokerHeartbeat * registerBroker (only for migrating ZK brokers) * unregisterBroker Raft events and other controller events do not follow the same code path as ControllerWriteEvent, so they are not affected by this new behavior. This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean MIGRATION rather than PRE_MIGRATION. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 007c0d375a6e70aefb65f58f9f096016c4cbea16 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 26 17:53:56 2023 +0530 KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634) Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local> Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com> commit baf127a6633161cb52747467880b006d2f54d3bd Author: Greg Harris <greg.harris@aiven.io> Date: Wed Apr 26 00:30:13 2023 -0700 KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429) Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io> commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef Author: Victoria Xia <victoria.xia@confluent.io> Date: Tue Apr 25 22:39:23 2023 -0400 KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622) Reviewers: Matthias J. Sax <matthias@confluent.io> commit a7d0b3f753708a93aea92e614833f6f6e7443234 Author: Said Boudjelda <bmscomp@gmail.com> Date: Tue Apr 25 23:31:04 2023 +0200 MINOR: Upgrade gradle to 8.1.1 (#13625) Also upgrade gradle plugins: - `org.owasp.dependencycheck` gradle plugin to version `8.2.1` - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1` Gradle release notes: * https://docs.gradle.org/8.1.1/release-notes.html Reviewers: Ismael Juma <ismael@juma.me.uk> commit 9a36da12b7359b7158332c541655716312efb5b3 Author: David Jacot <djacot@confluent.io> Date: Tue Apr 25 18:50:51 2023 +0200 KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538) This patch adds ConsumerGroupMember. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io> commit 4780dc773f2cd5a20fa5be38d20137745690a888 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue Apr 25 21:20:35 2023 +0530 KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com> commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4 Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com> Date: Tue Apr 25 13:28:37 2023 +0100 KAFKA-14592: Move FeatureCommand to tools (#13459) KAFKA-14592: Move FeatureCommand to tools Reviewers: Luke Chen <showuon@gmail.com> commit d83a734c41c43a03aa571e602cf8535f0893fd79 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 25 12:02:24 2023 +0200 MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626) The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call. Reviewers: maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com> commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 15:29:57 2023 -0700 KAFKA-12446: update change encoding to use varint (#13533) KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io> commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402 Author: Victoria Xia <victoria.xia@confluent.io> Date: Mon Apr 24 17:06:26 2023 -0400 KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615) Reviewers: Matthias J. Sax <matthias@confluent.io> commit 6dcdb017327587a6943fa868595fa3488c7f7ef7 Author: Matthias J. Sax <matthias@confluent.io> Date: Mon Apr 24 12:40:25 2023 -0700 KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592) Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks. This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors. Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io> commit 2271e748a11919d07698ebce759dca2e3075596a Author: Chia-Ping Tsai <chia7712@gmail.com> Date: Mon Apr 24 17:21:19 2023 +0800 MINOR: fix zookeeper_migration_test.py (#13620) Reviewers: Mickael Maison <mimaison@users.noreply.github.com> commit c3241236296f9aaee103e475f242e32f04d4c256 Author: Yash Mayya <yash.mayya@gmail.com> Date: Mon Apr 24 14:06:20 2023 +0530 KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 7061475445cb6314e7cf4f9848384224b14f4395 Author: Greg Harris <greg.harris@aiven.io> Date: Fri Apr 21 12:55:41 2023 -0700 KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575) Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest` Reviewers: Josep Prat <jlprat@apache.org> commit ecdef88f744410039b88e90a5979078d5735aa06 Author: Matthias J. Sax <matthias@confluent.io> Date: Fri Apr 21 12:48:05 2023 -0700 MINOR: updated KS release notes for 3.5 (#13577) Reviewers: Walker Carlson <wcarlson@confluent.io> commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55 Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Fri Apr 21 15:02:06 2023 +0200 MINOR: fix noticed typo in raft and metadata projects (#13612) Reviewers: Josep Prat <jlprat@apache.org> commit c39bf714bbaf2a632be4c2a7e446553fe40ba129 Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:22:16 2023 +0200 KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537) This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes. Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 2d0b816150c79057c813387bd126523b6326a1fc Author: David Jacot <djacot@confluent.io> Date: Fri Apr 21 11:19:04 2023 +0200 MINOR: Move `ControllerPurgatory` to `server-common` (#13555) This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit df137752542c005c6998c37c03222ffbeca0f349 Author: Purshotam Chauhan <pchauhan@confluent.io> Date: Fri Apr 21 14:08:23 2023 +0530 KAFKA-14828: Remove R/W locks using persistent data structures (#13437) Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only. In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation. Reviewers: Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Divij Vaidya <diviv@amazon.com> commit 2ee770ac7e576c85a911ccb307339be3d58a8942 Author: Colin P. McCabe <cmccabe@apache.org> Date: Thu Apr 20 10:21:26 2023 -0700 Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)" This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc. commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3 Author: Justine Olshan <jolshan@confluent.io> Date: Thu Apr 20 09:30:11 2023 -0700 KAFKA-14904: Pending state blocked verification of transactions (#13579) KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction. There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes. When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions. The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely. This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed. Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io> commit 7f175feacaba4187bbb3631dff3a1330060f191e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Thu Apr 20 13:50:29 2023 +0530 KAFKA-14586: Adding redirection for StreamsResetter (#13614) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com> commit e14dd8024adae746c50c8b7d9cd268e859669576 Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com> Date: Thu Apr 20 05:29:27 2023 +0200 KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432) We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic: 1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry. 2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of the client. 3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`. We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them. Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io> commit f5de4daa71f0ad31aa64443c5ff43c59712feea5 Author: Ron Dagostino <rndgstn@gmail.com> Date: Wed Apr 19 19:45:26 2023 -0400 KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change events. However, it does not distinguish between issues related to feature changes actually failing and other exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues. We shutdown the broker only for FeatureCacheUpdateException. Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 11c8bf4826197533807b2132cfc6599ba70de1c1 Author: Victoria Xia <victoria.xia@confluent.io> Date: Wed Apr 19 19:34:36 2023 -0400 KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609) This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter. Part of KIP-914. Reviewer: Matthias J. Sax <matthias@confluent.io> commit 809966a9a06664e0b521c2298fa0de834e443607 Author: Matthew de Detrich <matthew.dedetrich@aiven.io> Date: Wed Apr 19 20:54:07 2023 +0200 KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478) Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address. Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org> commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246 Author: David Arthur <mumrah@gmail.com> Date: Wed Apr 19 14:19:13 2023 -0400 KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606) This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit b10716e72370c4e128bddb17bcc107ccab221e47 Author: hudeqi <1217150961@qq.com> Date: Thu Apr 20 00:49:08 2023 +0800 KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com> commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc Author: Keith Wall <kwall@redhat.com> Date: Wed Apr 19 03:58:29 2023 +0100 KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572) Changes SocketServer to set the setReuseAddress(true) socket option. This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test. Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com> commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Apr 19 00:46:03 2023 +0530 MINOR: Fixing gradle build during compileScala and compileTestScala (#13588) Reviewers: Chia-Ping Tsai <chia7712@gmail.com> commit 3388adf1b52f545e2d69edd30cdd53241b2887f3 Author: Matthias J. Sax <matthias@confluent.io> Date: Tue Apr 18 11:32:27 2023 -0700 MINOR: rename internal FK-join processor classes (#13589) Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io> commit abca86511ecc9e081d676976ff6d3b845308f444 Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com> Date: Tue Apr 18 12:41:38 2023 -0400 KAFKA-14881: Rework UserScramCredentialRecord (#13513) Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This is necessary to support migration from ZK, since those are the fields we stored in ZK. Update latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version. Moved ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more appropriate. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 61530d68ce83467de6190a52da37b3c0af84f0ef Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 09:37:04 2023 -0400 KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526) This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields. Reviewers: David Jacot <djacot@confluent.io> commit b36a170aa3b2738177d7229859db765e0115385b Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com> Date: Tue Apr 18 13:36:56 2023 +0200 MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593) Reviewers: Luke Chen <showuon@gmail.com> commit 5e9d4de748dff7b91043a9b799716ab4becdae7e Author: Jeff Kim <kimkb2011@gmail.com> Date: Tue Apr 18 04:41:54 2023 -0400 KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511) This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io> commit 454b72161a76b1687a1263157d7cc30a1bdb2506 Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com> Date: Tue Apr 18 09:40:14 2023 +0200 KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557) KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow. To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread. commit e27926f92b1f6b34ed6731f33c712a5d0d594275 Author: Ron Dagostino <rndgstn@gmail.com> Date: Mon Apr 17 17:52:28 2023 -0400 KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io> commit 7159f6c1a81ead277030f55f293943270346ad4e Author: Alyssa Huang <ahuang@confluent.io> Date: Mon Apr 17 11:03:45 2023 -0700 MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it should have been. Add a regression test. Reviewers: Colin P. McCabe <cmccabe@apache.org> commit cc88822986e4004e772f376eae47dbe08e29d137 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed May 10 16:29:27 2023 -0700 small changes commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 25 14:58:20 2023 -0700 grammar and code comments changes commit 5550e4934ade28f069e4ae9ab97e76982ff3401f Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 17:39:48 2023 -0700 fixed formatting stuff commit 28ab29819bf327309ce07196a3fb0492f7fb2281 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 19 11:38:57 2023 -0700 consumers -> members commit e56ded267456bbe1c22ded0d037953545b921d1d Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 11:26:23 2023 -0700 Interface changes incorporated commit 2d50c278c18baa11b0235330149dd8f9def57c3d Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 13 14:47:33 2023 -0700 Renamed topicIdToPartition class commit d33bbfac053ccac77204ac4253698ed356d7c621 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 17:31:26 2023 -0700 Added stickiness test commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184 Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 13:28:24 2023 -0700 Separated uniform and general builder in different files, added more tests commit ef814867aba68fc4813359c5cef6ae155ed49a60 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:24:27 2023 -0700 import changes commit dffad677c6e4a33625401d621f046af6f21feb99 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:16:33 2023 -0700 Fixed checkstyle and added test file commit 7fd779418df3504a490c0d10f3848360a4417077 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 5 15:13:38 2023 -0700 First draft optimised uniform assignor code commit 309cc6141f7962e794c90d4107f460fafd51697d Merge: 3d9264547c 2c1cf03a89 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Mon Apr 17 11:03:07 2023 -0700 Merge branch 'trunk' into rreddy-22/KAFKA-14514 commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 10:58:16 2023 -0700 Interface changes incorporated, added stickiness test and non-existent topic test. commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436 Merge: a1bdb58e08 57b94ca208 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 17 09:49:23 2023 -0700 Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514 # Conflicts: # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java # group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java commit a1bdb58e082d98b73f50860db5b35684f4eba51b Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 20:01:59 2023 -0700 Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc. commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71 Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 11:28:40 2023 -0700 Addressed some PR comments commit 918d262ae52892c13766649db847ede6928921f8 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7369db3744b4c89f625f6b73db331eaf0512c4eb Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit a925b02dd3d78c8b9c1415c083f0544edcb404fe Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit 48a982232c6f1ff371cb44333fd12d6b37ef381a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Apr 12 13:00:07 2023 -0700 Removed topicIdToPartition class, changed attribute names and added java doc for getter methods commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Apr 11 15:09:44 2023 -0700 Made all attributes private and added getter methods, changed names according to PR comments commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8 Author: Ritika Reddy <rreddy@confluent.io> Date: Fri Apr 7 11:25:33 2023 -0700 Added toString method and hash commit 5065109332845541c5591a52683f21d37982d80a Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 13:43:26 2023 -0700 Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure commit 57b94ca208cb374b22d61eb044ef8004ea09479f Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 11:12:16 2023 -0700 Made subscribed topics a collection, removed * import exception commit 7352cbb5683f676382f56e6ae5c926f3cae98824 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Apr 6 10:55:06 2023 -0700 minor commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0 Merge: 4b321aa537 637bc92ba1 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Thu Apr 6 10:37:58 2023 -0700 Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514 commit 4b321aa5374427a15346be9b87b7c545ccb7db61 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Apr 3 11:54:59 2023 -0700 Addressed PR comments commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e Author: Ritika Reddy <rreddy@confluent.io> Date: Wed Mar 29 12:30:13 2023 -0700 Addressed PR comments commit 74105291cc687fe41b75de692fc624ab1335341a Author: Ritika Reddy <rreddy@confluent.io> Date: Tue Mar 28 12:21:33 2023 -0700 removed reduce partitions case commit 3df34605e078beb9babcbeadc7b119a70da63796 Author: Ritika Reddy <rreddy@confluent.io> Date: Mon Mar 27 20:17:14 2023 -0700 removed reduce partitions case commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328 Author: Ritika Reddy <rreddy@confluent.io> Date: Thu Mar 23 13:02:31 2023 -0700 Server Side Sticky Range Assignor full implementation

fvaleri added 6 commits March 30, 2023 11:47

Move all classes from kafka.examples to org.apache.kafka.examples

d1e4f70

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Avoid printing all message produced and consumed

c61b5f9

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Fix producer consumer demo haging on timeout

70e8932

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Remove constructor logic and use try-with-resources

943fd69

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Improve EOS example and processor

b3b8f0b

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

showuon reviewed Apr 5, 2023

View reviewed changes

fvaleri marked this pull request as draft April 6, 2023 09:20

fvaleri added a commit to fvaleri/kafka that referenced this pull request Apr 6, 2023

KAFKA-14752: Kafka examples improvements - consumer changes

5723a74

This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri marked this pull request as ready for review April 6, 2023 13:50

Restore original layout for review

5c9fea9

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri force-pushed the examples-imp branch from 6108771 to 5c9fea9 Compare April 13, 2023 13:58

fvaleri added 2 commits April 16, 2023 16:39

Improve error handling

58b54c3

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Add SerializationException

eb8914a

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri marked this pull request as draft May 2, 2023 12:57

fvaleri added a commit to fvaleri/kafka that referenced this pull request May 2, 2023

KAFKA-14752: Kafka examples improvements - producer changes

3d0c9cf

This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri closed this May 2, 2023

fvaleri deleted the examples-imp branch May 2, 2023 14:06

fvaleri added a commit to fvaleri/kafka that referenced this pull request May 9, 2023

KAFKA-14752: Kafka examples improvements - processor changes

8d45d79

This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri added a commit to fvaleri/kafka that referenced this pull request May 11, 2023

KAFKA-14752: Kafka examples improvements - demo changes

e6e2b3d

This is extracted from the original PR for better review. apache#13492 Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-14752: Kafka examples improvements #13492

KAFKA-14752: Kafka examples improvements #13492

fvaleri commented Apr 3, 2023

divijvaidya commented Apr 3, 2023

fvaleri commented Apr 3, 2023 •

edited

Loading

showuon left a comment •

edited

Loading

showuon Apr 5, 2023

fvaleri commented Apr 6, 2023 •

edited

Loading

fvaleri commented May 2, 2023 •

edited

Loading

KAFKA-14752: Kafka examples improvements #13492

KAFKA-14752: Kafka examples improvements #13492

Conversation

fvaleri commented Apr 3, 2023

divijvaidya commented Apr 3, 2023

fvaleri commented Apr 3, 2023 • edited Loading

showuon left a comment • edited Loading

Choose a reason for hiding this comment

showuon Apr 5, 2023

Choose a reason for hiding this comment

fvaleri commented Apr 6, 2023 • edited Loading

fvaleri commented May 2, 2023 • edited Loading

fvaleri commented Apr 3, 2023 •

edited

Loading

showuon left a comment •

edited

Loading

fvaleri commented Apr 6, 2023 •

edited

Loading

fvaleri commented May 2, 2023 •

edited

Loading