Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KAFKA-14834: [11/N] Update table joins to identify out-of-order records with isLatest #13609

Merged
merged 4 commits into from Apr 19, 2023
Merged

KAFKA-14834: [11/N] Update table joins to identify out-of-order records with isLatest #13609

merged 4 commits into from Apr 19, 2023

Conversation

vcrfxia
Copy link
Collaborator

@vcrfxia vcrfxia commented Apr 18, 2023 •
edited

There's a bug in the table-table join handling of out-of-order records in versioned tables (see #13510 and #13522 for context) where if the latest value for a particular key is a tombstone, then out-of-order records are not properly identified because versioned stores do not return timestamps for tombstones (so there is no timestamp to compare against, when deciding whether a record is out-of-order or not). This results in out-of-order records improperly being identified as not out-of-order, when the latest value for the key is a tombstone.

This PR fixes the bug by using the isLatest value from the Change object (see #13564) instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter. This also enables us to remove the additional state store access granted to join processors in #13510 and #13522, resulting in a cleaner topology.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

vcrfxia
vcrfxia commented Apr 18, 2023
View reviewed changes
streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableImpl.java
@@ -804,6 +806,7 @@ private <VO, VR> KTable<K, VR> doJoin(final KTable<K, VO> other,
kTableKTableJoinNode.setOutputVersioned(isOutputVersioned);

builder.addGraphNode(this.graphNode, kTableKTableJoinNode);
builder.addGraphNode(((KTableImpl<?, ?, ?>) other).graphNode, kTableKTableJoinNode);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know why it's currently the case that primary-key table-table join nodes only have one parent, instead of two. Seems more correct to have two, and the GraphNode mechanism for determining whether the joining table is versioned or not will not work without this parent connection.

I have verified that there is no change to the built topology, so AFAICT this addition is internal-only.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Seems to be incorrect, but did apparently not surface as a bug. Nice fix!

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableImpl.java
@@ -1098,7 +1101,7 @@ private <VR, KO, VO> KTable<K, VR> doJoinOnForeignKey(final KTable<KO, VO> forei
//not be done needlessly.
((KTableImpl<?, ?, ?>) foreignKeyTable).enableSendingOldValues(true);

//Old values must be sent such that the ForeignJoinSubscriptionSendProcessorSupplier can propagate deletions to the correct node.
//Old values must be sent such that the SubscriptionSendProcessorSupplier can propagate deletions to the correct node.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This, and a few other similar renames in comments, are unrelated to this PR but included as cleanup from #13589.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! -- Could we do a small follow up for 3.4 branch to get it backported?

Copy link
Collaborator Author

@vcrfxia vcrfxia Apr 19, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing: #13610

vcrfxia
vcrfxia commented Apr 18, 2023
View reviewed changes
streams/src/test/java/org/apache/kafka/streams/integration/AbstractJoinIntegrationTest.java
// the result, and the main `input` list results in too many result records/test noise.
// also used for table-table multi-join tests, since out-of-order data with table-table
// joins is already tested in non-multi-join settings.
protected final List<Input<String>> inputWithoutOutOfOrderData = Arrays.asList(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As suggested in #13497 (comment), I have split the input test data into two copies: a smaller one without out-of-order data (for use in stream-stream join and table-table multi-join tests) and a larger one with out-of-order data (for use elsewhere, including to validate versioned joins).

@vcrfxia vcrfxia mentioned this pull request Apr 18, 2023
Merged
3 tasks
@mjsax mjsax added streams kip Requires or implements a KIP labels Apr 18, 2023
mjsax
mjsax reviewed Apr 19, 2023
View reviewed changes
streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableImpl.java
@@ -804,6 +806,7 @@ private <VO, VR> KTable<K, VR> doJoin(final KTable<K, VO> other,
kTableKTableJoinNode.setOutputVersioned(isOutputVersioned);

builder.addGraphNode(this.graphNode, kTableKTableJoinNode);
builder.addGraphNode(((KTableImpl<?, ?, ?>) other).graphNode, kTableKTableJoinNode);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Seems to be incorrect, but did apparently not surface as a bug. Nice fix!

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableImpl.java
@@ -1098,7 +1101,7 @@ private <VR, KO, VO> KTable<K, VR> doJoinOnForeignKey(final KTable<KO, VO> forei
//not be done needlessly.
((KTableImpl<?, ?, ?>) foreignKeyTable).enableSendingOldValues(true);

//Old values must be sent such that the ForeignJoinSubscriptionSendProcessorSupplier can propagate deletions to the correct node.
//Old values must be sent such that the SubscriptionSendProcessorSupplier can propagate deletions to the correct node.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! -- Could we do a small follow up for 3.4 branch to get it backported?

...ams/src/main/java/org/apache/kafka/streams/kstream/internals/graph/KTableKTableJoinNode.java
}
final KTableKTableAbstractJoin<K, ?, ?, ?> tableJoin = (KTableKTableAbstractJoin<K, ?, ?, ?>) processorSupplier;

if (parentNodeName.equals(tableJoin.joinThisParentNodeName())) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I understand this condition? Can you elaborate? It seems to be the only place when we sue the newly added parentNodeName -- why do we not use it elsewhere?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question -- the answer is that this join node has multiple parents, while all the other VersionedSemanticsGraphNode implementations only have a single parent. Because this join node has multiple parents and writes two separate processors (one for each side of the join) to the topology, when enabling versioned semantics we need a way to distinguish which side of the join we're enabling versioned semantics for. In other words, it's possible that only one side of the join is "versioned," in which case one of the two join processors should have versioned semantics enabled while the other should not. The way that we determine which side of the join to enable versioned semantics for is based on the parent node name; the processor whose "joinThis" is the parent node which has been identified as versioned is the processor for which versioned semantics will be enabled. (In the case of a self-join, both processors will satisfy this check, and both processors will have versioned semantics enabled.)

For all other VersionedSemanticsGraphNode implementations which only have a single parent, we could also perform an analogous parent node name check if we wanted to, but the parent node name should always match so it'd be redundant.

streams/src/test/java/org/apache/kafka/streams/integration/TableTableJoinIntegrationTest.java Outdated
null,
null,
null,
null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It think there is one null line too many?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, good eye. Fixed.

streams/src/test/java/org/apache/kafka/streams/integration/TableTableJoinIntegrationTest.java Outdated
Collections.singletonList(new TestRecord<>(ANY_UNIQUE_KEY, null, null, 14L)),
null,
null,
Collections.singletonList(new TestRecord<>(ANY_UNIQUE_KEY, "F-e", null, 14L))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the last two result row flip: we first get F-e when we process left hand F and get nothing when we process right hand f?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are correct. I've fixed this and also strengthened the check that the test uses (in the latest commit) so that this type of "error" fails the test now. This required updating some of the table-table multi-join expected test results in the process, separate from versioned joins.

@vcrfxia vcrfxia mentioned this pull request Apr 19, 2023
Merged
3 tasks
@mjsax mjsax merged commit 11c8bf4 into apache:trunk Apr 19, 2023
1 check failed
mjsax pushed a commit that referenced this pull request Apr 19, 2023
@vcrfxia @mjsax
KAFKA-14834: [11/N] Update table joins to identify out-of-order recor…
6cf1010 
…ds with `isLatest` (#13609)

This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter.

Part of KIP-914.

Reviewer: Matthias J. Sax <matthias@confluent.io>
@mjsax
Copy link
Member

mjsax commented Apr 19, 2023

Merged to trunk and cherry-picked to 3.5 branch.

@vcrfxia vcrfxia deleted the kip-914-table-join-fixes branch April 20, 2023 00:03
rreddy-22 added a commit to rreddy-22/kafka-rreddy that referenced this pull request Jun 29, 2023
@rreddy-22
Squashed commit of the following:
1c2475b 
commit c757af5
Author: Federico Valeri <fedevaleri@gmail.com>
Date:   Fri May 12 04:39:12 2023 +0200

    KAFKA-14752: Kafka examples improvements - demo changes (apache#13517)

    KAFKA-14752: Kafka examples improvements - demo changes

    Reviewers: Luke Chen <showuon@gmail.com>

commit 54a4067
Author: Kamal Chandraprakash <kchandraprakash@uber.com>
Date:   Thu May 11 21:49:21 2023 +0530

    KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (apache#13060)

    Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org>

commit bd65db8
Author: Josep Prat <josep.prat@aiven.io>
Date:   Thu May 11 17:55:26 2023 +0200

    MINOR: clean up unused methods in core utils (apache#13706)

    Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org>

commit 38adb39
Author: Jeff Kim <kimkb2011@gmail.com>
Date:   Thu May 11 07:51:41 2023 -0400

    MINOR: add test tag for testDeadToDeadIllegalTransition (apache#13694)

    Reviewers: David Jacot <djacot@confluent.io>

commit ee41328
Author: Federico Valeri <fedevaleri@gmail.com>
Date:   Thu May 11 11:19:32 2023 +0200

    KAFKA-14752: Kafka examples improvements - processor changes (apache#13516)

    Reviewers: Luke Chen <showuon@gmail.com>

commit a263627
Author: Mickael Maison <mimaison@users.noreply.github.com>
Date:   Thu May 11 11:02:45 2023 +0200

    MINOR: Remove unused methods in CoreUtils (apache#13170)

    Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com>

commit 920a360
Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
Date:   Thu May 11 04:13:29 2023 +0200

    MINOR: fix a small typo in SharedServer.scala (apache#13693)

    Diabled -> Disabled

commit 6d2ad4a
Author: A. Sophie Blee-Goldman <sophie@confluent.io>
Date:   Wed May 10 13:39:15 2023 -0700

    HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (apache#13695)

    The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen()

    Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io>

commit f17fb75
Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
Date:   Wed May 10 16:41:52 2023 +0200

    KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (apache#13690)

    Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

commit 4653507
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Wed May 10 05:09:12 2023 -0700

    KAFKA-14514; Add Range Assignor on the Server (KIP-848) (apache#13443)

    This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers.

    Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io>

commit 625ef17
Author: Luke Chen <showuon@gmail.com>
Date:   Wed May 10 16:40:20 2023 +0800

    MINOR: remove kraft readme link (apache#13691)

    The config/kraft/README.md is already removed. We should also remove the link.

    Reviewers: dengziming <dengziming1993@gmail.com>

commit 228434d
Author: Jeff Kim <kimkb2011@gmail.com>
Date:   Tue May 9 10:49:27 2023 -0400

    KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (apache#13644)

    This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java.

    Reviewers: David Jacot <djacot@confluent.io>

commit 59ba9db
Author: Yash Mayya <yash.mayya@gmail.com>
Date:   Tue May 9 17:58:45 2023 +0530

    KAFKA-14974: Restore backward compatibility in KafkaBasedLog (apache#13688)

    `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods.

    This refactoring changes no functionality other than restoring the older methods.

    Reviewers: Randall Hauch <rhauch@gmail.com>

commit b40a7fc
Author: Matthias J. Sax <matthias@confluent.io>
Date:   Mon May 8 14:24:11 2023 -0700

    HOTFIX: fix broken Streams upgrade system test (apache#13654)

    Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io>

commit 7634eee
Author: David Jacot <djacot@confluent.io>
Date:   Mon May 8 20:46:07 2023 +0200

    KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (apache#13638)

    This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state.

    Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

commit 86daf8c
Author: vamossagar12 <sagarmeansocean@gmail.com>
Date:   Mon May 8 20:09:47 2023 +0530

    KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (apache#13594)

    apache#13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR.

    Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places.

    ---------

    Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

    Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

commit 2b98f85
Author: Christo Lolov <lolovc@amazon.com>
Date:   Mon May 8 15:24:52 2023 +0100

    KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (apache#13621)

    Migrates ChangeLogReader mock in TaskManagerTest to mockito.

    Reviewer: Bruno Cadonna <cadonna@apache.org>

commit 3472389
Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
Date:   Mon May 8 07:36:36 2023 +0100

    KAFKA-14662: Update the ACL list in the doc (apache#13660)

    Added the missing ACLs to the doc.

    Reviewers: Luke Chen <showuon@gmail.com>

commit a556def
Author: Divij Vaidya <diviv@amazon.com>
Date:   Mon May 8 08:35:14 2023 +0200

    MINOR: Print the cause of failure for test (apache#13672)

    Motivation

    PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build):

    org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
    	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
    	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
    	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
    	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
    	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
    	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228)
    	at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is.
    Change

        1. Make the test a bit more robust by waiting for server startup.
        2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure.

    Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

commit 2607e0e
Author: Federico Valeri <fedevaleri@gmail.com>
Date:   Mon May 8 08:33:31 2023 +0200

    MINOR: Fix producer Callback comment (apache#13669)

    Fixes the wrong exception name: OffsetMetadataTooLargeException.

    Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com>

commit 78090bb
Author: Federico Valeri <fedevaleri@gmail.com>
Date:   Mon May 8 04:15:52 2023 +0200

    KAFKA-14752: Kafka examples improvements - producer changes (apache#13515)

    KAFKA-14752: Kafka examples improvements - producer changes

    Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com>

commit 6e7144a
Author: Chia-Ping Tsai <chia7712@gmail.com>
Date:   Sat May 6 02:56:26 2023 +0800

    MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (apache#13659)

    Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro>

commit 6bcc497
Author: Divij Vaidya <diviv@amazon.com>
Date:   Fri May 5 14:05:20 2023 +0200

    KAFKA-14766: Improve performance of VarInt encoding and decoding (apache#13312)

    Motivation

    Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit.

    In this PR, we only update varInt32 encoding/decoding.
    Changes

    This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation.
    Results

    Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java

    Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com>

commit e34f884
Author: Divij Vaidya <diviv@amazon.com>
Date:   Fri May 5 13:55:17 2023 +0200

    KAFKA-14926: Remove metrics on Log Cleaner shutdown (apache#13623)

    When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective.

    Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io>

commit 0822ce0
Author: David Arthur <mumrah@gmail.com>
Date:   Fri May 5 04:35:26 2023 -0400

    KAFKA-14840: Support for snapshots during ZK migration (apache#13461)

    This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active
    controller loaded a snapshot, it would get out of sync with the ZK state.

    In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to
    compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods
    from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and
    AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass
    and without using too much memory.

    Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com>

commit 97c36f3
Author: Colin P. McCabe <cmccabe@apache.org>
Date:   Thu May 4 12:20:33 2023 -0700

    HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging apache#13540

commit 63f9f23
Author: Colin P. McCabe <cmccabe@apache.org>
Date:   Thu May 4 11:18:03 2023 -0700

    MINOR: improve QuorumController logging apache#13540

    When creating the QuorumController, log whether ZK migration is enabled.

    When applying a feature level record which sets the metadata version, log the metadata version enum
    rather than the numeric feature level.

    Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the
    end of replay.

    When TRACE is enabled, log every record that is replayed in QuorumController. Since some records
    may contain sensitive information, create RecordRedactor to assist in logging only what is safe to
    put in the log4j file.

    Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged
    at INFO, and additions are logged at TRACE.

    Remove SnapshotReason.java, SnapshotReasonTest.java, and
    QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to
    org.apache.kafka.image.publisher.SnapshotGenerator.

    Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org>

commit ffd814d
Author: Justine Olshan <jolshan@confluent.io>
Date:   Thu May 4 09:55:45 2023 -0700

    KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (apache#13607)

    Also modifies verification to only add a partition to verify if it is transactional.

    When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different.

    Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

commit ea81e99
Author: Philip Nee <pnee@confluent.io>
Date:   Thu May 4 09:20:01 2023 -0700

    KAFKA-13668: Retry upon missing initProducerId due to authorization error (apache#12149)

    Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer.

    This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin.

    Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest.

    Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>

commit dc7819d
Author: Christo Lolov <lolovc@amazon.com>
Date:   Thu May 4 11:00:33 2023 +0100

    KAFKA-14594: Move LogDirsCommand to tools module (apache#13122)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>

commit d46c3f2
Author: David Mao <47232755+splett2@users.noreply.github.com>
Date:   Wed May 3 17:09:43 2023 -0700

    MINOR: Reduce number of threads created for integration test brokers (apache#13655)

    The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker.

    Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io>

commit c08120f
Author: Jason Gustafson <jason@confluent.io>
Date:   Wed May 3 15:25:32 2023 -0700

    MINOR: Allow tagged fields with version subset of flexible version range (apache#13551)

    The generated message types are missing a range check for the case when the tagged version range is a subset of
    the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the
    number of tags serialized.

    Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit b620c03
Author: Luke Chen <showuon@gmail.com>
Date:   Thu May 4 01:08:25 2023 +0800

    KAFKA-14946: fix NPE when merging the deltatable (apache#13653)

    Fix NPE while merging the deltatable. Because it's possible that hashTier is
    not null but deltatable is null (ex: removing data), we should have null check
    while merging for deltatable like other places did. Also added tests that will
    fail without this change.

    Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit 4a0b6eb
Author: Yash Mayya <yash.mayya@gmail.com>
Date:   Tue May 2 23:16:46 2023 +0530

    KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (apache#13657)

    Reviewers: Chris Egerton <chrise@aiven.io>

commit 16fc8e1
Author: David Jacot <djacot@confluent.io>
Date:   Tue May 2 18:04:50 2023 +0200

    KAFKA-14462; [10/N] Add TargetAssignmentBuilder (apache#13637)

    This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group.

    Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

commit f44ee4f
Author: Christo Lolov <lolovc@amazon.com>
Date:   Tue May 2 16:39:31 2023 +0100

    MINOR: Remove unnecessary code in client/connect (apache#13259)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>

commit 33012b5
Author: Federico Valeri <fedevaleri@gmail.com>
Date:   Tue May 2 14:28:42 2023 +0200

    KAFKA-14752: Kafka examples improvements - consumer changes (apache#13514)

    KAFKA-14752: Kafka examples improvements - consumer changes

    This is extracted from the original PR for better review.
    apache#13492

    Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

    Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>

commit 141c76a
Author: Bruno Cadonna <cadonna@apache.org>
Date:   Tue May 2 14:00:34 2023 +0200

    KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (apache#13529)

    1. Migrates topology builder mock in TaskManagerTest to mockito.

    2. Replaces the unit test to verify if subscribed partitions are added
    to topology metadata.

    3. Modifies signatures of methods for adding subscribed partitions to
    topology metadata to use sets instead of lists. This makes the
    intent of the methods clearer and makes the tests more portable.

    Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org>

commit 21af191
Author: Luke Chen <showuon@gmail.com>
Date:   Tue May 2 09:54:12 2023 +0800

    MINOR: Add reason to exceptions in QuorumController (apache#13648)

    Saw this error message in log:

    ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController)

    The null reason is not helpful with only RejectedExecutionException. Adding the reason to it.

    Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

commit 4773961
Author: Yash Mayya <yash.mayya@gmail.com>
Date:   Mon May 1 21:51:09 2023 +0530

    MINOR: Fix Javadoc for configureAdminResources  in Connect's RestServer (apache#13635)

    Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io>

commit e299423
Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
Date:   Mon May 1 09:56:04 2023 -0400

    KAFKA-14859: SCRAM ZK to KRaft migration with dual write (apache#13628)

    Handle migrating SCRAM records in ZK when migrating from ZK to KRaft.

    This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes.

    Reviewers: David Arthur <mumrah@gmail.com>

commit 64ebbc5
Author: Philip Nee <pnee@confluent.io>
Date:   Fri Apr 28 08:46:40 2023 -0700

    MINOR: Fixing typos in the ConsumerCoordinator (apache#13618)

    Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io>

commit e55fbce
Author: Anton Agestam <anton.agestam@aiven.io>
Date:   Fri Apr 28 11:54:28 2023 +0100

    MINOR: Fix incorrect description of SessionLifetimeMs (apache#13649)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>

commit c6ad151
Author: Philip Nee <pnee@confluent.io>
Date:   Fri Apr 28 02:08:32 2023 -0700

    KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (apache#13550)

    This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS.

    This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method.  In short, it should no long check and validate only the highest generation.  Instead, we consider 3 cases:
    1. Member will continue to hold on to its partition if there are no other owners
    2. If there are 1+ owners to the same partition. One with the highest generation will win.
    3. If two members of the same generation hold on to the same partition.  We will log an error but remove both from the assignment. (Same with the current logic)

    Here are some important notes that lead to the patch:
    - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown.
    - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error.  This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive

    To summarize the current behavior of different errors:
    `REBALANCE_IN_PROGRESS`
    - heartbeat: requestRejoin if member state is stable
    - joinGroup: rejoin immediately
    - syncGroup: rejoin immediately
    - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress.

    `UNKNOWN_MEMBER_ID`
    - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
    - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately
    - syncGroup:  resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

    `ILLEGAL_GENERATION`
    - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
    - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing.  then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

    Reviewers: David Jacot <djacot@confluent.io>

commit 10b3e66
Author: Federico Valeri <fedevaleri@gmail.com>
Date:   Fri Apr 28 10:32:11 2023 +0200

    KAFKA-14584: Deprecate StateChangeLogMerger tool (apache#13171)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>

commit d796480
Author: Luke Chen <showuon@gmail.com>
Date:   Fri Apr 28 14:35:12 2023 +0800

    KAFKA-14909: check zkMigrationReady tag before migration (apache#13631)

    1. add ZkMigrationReady in apiVersionsResponse
    2. check all nodes if ZkMigrationReady are ready before moving to next migration state

    Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com>

commit c708f7b
Author: Colin Patrick McCabe <cmccabe@apache.org>
Date:   Thu Apr 27 19:15:26 2023 -0700

    MINOR: remove spurious call to fatalFaultHandler (apache#13651)

    Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805.  We should only
    invoke the fatal fault handller if we are unable to generate the activation records. If we are
    unable to write the activation records, a controller failover should be sufficient to remedy the
    situation.

    Co-authored-by: Luke Chen showuon@gmail.com

    Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com>

commit 056657d
Author: David Arthur <mumrah@gmail.com>
Date:   Thu Apr 27 17:04:56 2023 -0400

    MINOR add license to reviewers.py

commit a08f31e
Author: David Arthur <mumrah@gmail.com>
Date:   Thu Apr 27 14:32:19 2023 -0400

    MINOR: Adding reviewers.py (apache#11096)

    This script can be used to help build the "Reviewers: " string we include in commit messages.

    Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org>

commit 7049333
Author: Colin P. McCabe <cmccabe@apache.org>
Date:   Wed Apr 26 16:10:46 2023 -0700

    KAFKA-14943: Fix ClientQuotaControlManager validation

    Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be
    used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of
    internal ZK representation details that treated them both as "client configs.")

    Add unit tests for ClientQuotaControlManager.isValidIpEntity and
    ClientQuotaControlManager.configKeysForEntityType.

    This change doesn't affect metadata record application, only input validation. If there are bad
    client quotas that are set currently, this change will not alter the current behavior (of throwing
    an exception and ignoring the bad quota).

commit 8bde4e7
Author: David Jacot <djacot@confluent.io>
Date:   Thu Apr 27 14:05:41 2023 +0200

    KAFKA-14462; [9/N] Add RecordHelpers (apache#13544)

    This patch adds RecordHelpers.

    Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

commit dd6690a
Author: LinShunKang <linshunkang.chn@gmail.com>
Date:   Thu Apr 27 10:44:08 2023 +0800

    KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (apache#12545)

    This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
    Direct use ByteBuffer instead of byte[] to deserialize.

    Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro>

commit c1b5c75
Author: David Arthur <mumrah@gmail.com>
Date:   Wed Apr 26 10:20:30 2023 -0400

    KAFKA-14805 KRaft controller supports pre-migration mode (apache#13407)

    This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode,
    the controller will only allow certain write operations. The purpose of this is to disallow metadata
    changes when the controller is waiting for the ZK migration records to be committed.

    The following ControllerWriteEvent operations are permitted in pre-migration mode

    * completeActivation
    * maybeFenceReplicas
    * writeNoOpRecord
    * processBrokerHeartbeat
    * registerBroker (only for migrating ZK brokers)
    * unregisterBroker

    Raft events and other controller events do not follow the same code path as ControllerWriteEvent,
    so they are not affected by this new behavior.

    This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState

    In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean
    MIGRATION rather than PRE_MIGRATION.

    Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

commit 007c0d3
Author: vamossagar12 <sagarmeansocean@gmail.com>
Date:   Wed Apr 26 17:53:56 2023 +0530

    KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (apache#13634)

    Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

    Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

commit baf127a
Author: Greg Harris <greg.harris@aiven.io>
Date:   Wed Apr 26 00:30:13 2023 -0700

    KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (apache#13429)

    Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io>

commit ced1f62
Author: Victoria Xia <victoria.xia@confluent.io>
Date:   Tue Apr 25 22:39:23 2023 -0400

    KAFKA-14834: [13/N] Docs updates for versioned store semantics (apache#13622)

    Reviewers: Matthias J. Sax <matthias@confluent.io>

commit a7d0b3f
Author: Said Boudjelda <bmscomp@gmail.com>
Date:   Tue Apr 25 23:31:04 2023 +0200

    MINOR: Upgrade gradle to 8.1.1 (apache#13625)

    Also upgrade gradle plugins:
     - `org.owasp.dependencycheck` gradle plugin to version `8.2.1`
     - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1`

    Gradle release notes:
    * https://docs.gradle.org/8.1.1/release-notes.html

    Reviewers: Ismael Juma <ismael@juma.me.uk>

commit 9a36da1
Author: David Jacot <djacot@confluent.io>
Date:   Tue Apr 25 18:50:51 2023 +0200

    KAFKA-14462; [8/N] Add ConsumerGroupMember (apache#13538)

    This patch adds ConsumerGroupMember.

    Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io>

commit 4780dc7
Author: Yash Mayya <yash.mayya@gmail.com>
Date:   Tue Apr 25 21:20:35 2023 +0530

    KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (apache#13636)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

commit ea540fa
Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
Date:   Tue Apr 25 13:28:37 2023 +0100

    KAFKA-14592: Move FeatureCommand to tools (apache#13459)

    KAFKA-14592: Move FeatureCommand to tools

    Reviewers: Luke Chen <showuon@gmail.com>

commit d83a734
Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
Date:   Tue Apr 25 12:02:24 2023 +0200

    MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (apache#13626)

    The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation
    in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call.

    Reviewers:   maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com>

commit 2557a4b
Author: Matthias J. Sax <matthias@confluent.io>
Date:   Mon Apr 24 15:29:57 2023 -0700

    KAFKA-12446: update change encoding to use varint (apache#13533)

    KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead.

    Reviewers: Victoria Xia <victoria.xia@confluent.io>,  Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io>

commit ab8f285
Author: Victoria Xia <victoria.xia@confluent.io>
Date:   Mon Apr 24 17:06:26 2023 -0400

    KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (apache#13615)

    Reviewers: Matthias J. Sax <matthias@confluent.io>

commit 6dcdb01
Author: Matthias J. Sax <matthias@confluent.io>
Date:   Mon Apr 24 12:40:25 2023 -0700

    KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (apache#13592)

    Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks.

    This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors.

    Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io>

commit 2271e74
Author: Chia-Ping Tsai <chia7712@gmail.com>
Date:   Mon Apr 24 17:21:19 2023 +0800

    MINOR: fix zookeeper_migration_test.py (apache#13620)

    Reviewers: Mickael Maison <mimaison@users.noreply.github.com>

commit c324123
Author: Yash Mayya <yash.mayya@gmail.com>
Date:   Mon Apr 24 14:06:20 2023 +0530

    KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (apache#13587)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>

commit 7061475
Author: Greg Harris <greg.harris@aiven.io>
Date:   Fri Apr 21 12:55:41 2023 -0700

    KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (apache#13575)

    Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest`

    Reviewers: Josep Prat <jlprat@apache.org>

commit ecdef88
Author: Matthias J. Sax <matthias@confluent.io>
Date:   Fri Apr 21 12:48:05 2023 -0700

    MINOR: updated KS release notes for 3.5 (apache#13577)

    Reviewers: Walker Carlson <wcarlson@confluent.io>

commit dd63d88
Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
Date:   Fri Apr 21 15:02:06 2023 +0200

    MINOR: fix noticed typo in raft and metadata projects (apache#13612)

    Reviewers: Josep Prat <jlprat@apache.org>

commit c39bf71
Author: David Jacot <djacot@confluent.io>
Date:   Fri Apr 21 11:22:16 2023 +0200

    KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (apache#13537)

    This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes.

    Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

commit 2d0b816
Author: David Jacot <djacot@confluent.io>
Date:   Fri Apr 21 11:19:04 2023 +0200

    MINOR: Move `ControllerPurgatory` to `server-common` (apache#13555)

    This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module.

    Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org>

commit df13775
Author: Purshotam Chauhan <pchauhan@confluent.io>
Date:   Fri Apr 21 14:08:23 2023 +0530

    KAFKA-14828: Remove R/W locks using persistent data structures (apache#13437)

    Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only.

    In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation.

    Reviewers:  Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>,  Divij Vaidya <diviv@amazon.com>

commit 2ee770a
Author: Colin P. McCabe <cmccabe@apache.org>
Date:   Thu Apr 20 10:21:26 2023 -0700

    Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (apache#13572)"

    This reverts commit d04c3e5.

commit ef09a2e
Author: Justine Olshan <jolshan@confluent.io>
Date:   Thu Apr 20 09:30:11 2023 -0700

    KAFKA-14904: Pending state blocked verification of transactions (apache#13579)

    KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction.

    There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes.

    When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions.

    The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely.

    This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed.

    Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

commit 7f175fe
Author: vamossagar12 <sagarmeansocean@gmail.com>
Date:   Thu Apr 20 13:50:29 2023 +0530

    KAFKA-14586: Adding redirection for StreamsResetter (apache#13614)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

commit e14dd80
Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com>
Date:   Thu Apr 20 05:29:27 2023 +0200

    KAFKA-14821 Implement the listOffsets API with AdminApiDriver (apache#13432)

    We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic:
    1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry.
    2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of  the client.
    3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`.

    We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them.

    Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io>

commit f5de4da
Author: Ron Dagostino <rndgstn@gmail.com>
Date:   Wed Apr 19 19:45:26 2023 -0400

    KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires

    FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change
    events. However, it does not distinguish between issues related to feature changes actually failing and other
    exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration
    could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between
    these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues.
    We shutdown the broker only for FeatureCacheUpdateException.

    Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

commit 11c8bf4
Author: Victoria Xia <victoria.xia@confluent.io>
Date:   Wed Apr 19 19:34:36 2023 -0400

    KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (apache#13609)

    This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter.

    Part of KIP-914.

    Reviewer: Matthias J. Sax <matthias@confluent.io>

commit 809966a
Author: Matthew de Detrich <matthew.dedetrich@aiven.io>
Date:   Wed Apr 19 20:54:07 2023 +0200

    KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (apache#11478)

    Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address.

    Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org>

commit 750cfd8
Author: David Arthur <mumrah@gmail.com>
Date:   Wed Apr 19 14:19:13 2023 -0400

    KAFKA-14918 Only send controller RPCs to migrating ZK brokers (apache#13606)

    This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers.

    Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit b10716e
Author: hudeqi <1217150961@qq.com>
Date:   Thu Apr 20 00:49:08 2023 +0800

    KAFKA-14868: Remove all ReplicaManager metrics when it is closed (apache#13471)

    Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com>

commit d04c3e5
Author: Keith Wall <kwall@redhat.com>
Date:   Wed Apr 19 03:58:29 2023 +0100

    KAFKA-14908: Set setReuseAddress on the kafka server socket (apache#13572)

    Changes SocketServer to set the setReuseAddress(true) socket option.

    This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test.

    Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com>

commit f905a5a
Author: vamossagar12 <sagarmeansocean@gmail.com>
Date:   Wed Apr 19 00:46:03 2023 +0530

    MINOR: Fixing gradle build during compileScala and compileTestScala (apache#13588)

    Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

commit 3388adf
Author: Matthias J. Sax <matthias@confluent.io>
Date:   Tue Apr 18 11:32:27 2023 -0700

    MINOR: rename internal FK-join processor classes (apache#13589)

    Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>

commit abca865
Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
Date:   Tue Apr 18 12:41:38 2023 -0400

    KAFKA-14881: Rework UserScramCredentialRecord (apache#13513)

    Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This
    is necessary to support migration from ZK, since those are the fields we stored in ZK.  Update
    latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version.  Moved
    ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more
    appropriate.

    Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit 61530d6
Author: Jeff Kim <kimkb2011@gmail.com>
Date:   Tue Apr 18 09:37:04 2023 -0400

    KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (apache#13526)

    This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields.

    Reviewers: David Jacot <djacot@confluent.io>

commit b36a170
Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
Date:   Tue Apr 18 13:36:56 2023 +0200

    MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (apache#13593)

    Reviewers: Luke Chen <showuon@gmail.com>

commit 5e9d4de
Author: Jeff Kim <kimkb2011@gmail.com>
Date:   Tue Apr 18 04:41:54 2023 -0400

    KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (apache#13511)

    This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types.

    Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io>

commit 454b721
Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
Date:   Tue Apr 18 09:40:14 2023 +0200

    KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (apache#13557)

    KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow.
    To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread.

commit e27926f
Author: Ron Dagostino <rndgstn@gmail.com>
Date:   Mon Apr 17 17:52:28 2023 -0400

    KAFKA-14735: Improve KRaft metadata image change performance at high … (apache#13280)

    topic counts.

    Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change.  Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events.  We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods.

    Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>

commit 7159f6c
Author: Alyssa Huang <ahuang@confluent.io>
Date:   Mon Apr 17 11:03:45 2023 -0700

    MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields

    Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it
    should have been. Add a regression test.

    Reviewers: Colin P. McCabe <cmccabe@apache.org>
rreddy-22 added a commit to rreddy-22/kafka-rreddy that referenced this pull request Jul 24, 2023
@rreddy-22
Squashed commit of the following:
4f3c713 
commit 061dac797bae56d91659fbc8d7900117eb10a866
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:43:12 2023 -0700

    Moved SubscribedTopicMetadata to the consumer package

commit 6650ea5cd707f03713fc2c69ba76fafd56d886b8
Merge: 22f2df02de faafed25a1
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:33:17 2023 -0700

    Merge remote-tracking branch 'origin/rreddy-22/Rack-Assignor-Interface-Changes' into rreddy-22/Rack-Assignor-Interface-Changes

commit 22f2df02deabe1a6b5b8b8cceb432486b9ef4470
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:25:11 2023 -0700

    minor changes

commit faafed25a10d7ff79fb8a24ab62aa222c0e6516a
Merge: 4511683d4e 1bf73d89d0
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Fri Jul 21 12:50:59 2023 -0700

    Merge branch 'trunk' into rreddy-22/Rack-Assignor-Interface-Changes

commit 4511683d4ef489cb979769ff6775dfe0250cdeeb
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 21 03:09:27 2023 -0700

    Changes based on PR comments

commit 579bb948fcc78afd163410a5bc43832bc87aaf71
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Jul 19 14:45:18 2023 -0700

    New interfaces and classes added to facilitate passing rack information to the assignor, modified tests to incorporate changes

commit f68471a99f2e0260b36047a7596b0f9de48794ad
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Jul 13 11:55:20 2023 -0700

    Small edits

commit c50f3bdbf79f1ca127355d2a9002b1fdbf0baa2f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 11 16:30:32 2023 -0700

    Removed extra lines

commit 478488e0943aa879b27eec5ae5eb79a23dd9cab3
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 11 16:21:54 2023 -0700

    Added rack interface changes, added a new TopicAndClusterMetadata class to handle topic and cluster images and added an AbstractPartitionAssignor

commit ce540f943b8dae0786a87e51cafacb66efd1e64d
Merge: 9815d6db04 fd5b300b57
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:45:27 2023 -0600

    Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized

commit 9815d6db04c264456b7ead7fc5f735513a868c7d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:23:00 2023 -0600

    Abstract Partition Assignor

commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Jun 29 16:36:53 2023 -0700

    Refactored code to reduce format conversion time

commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 31 16:21:01 2023 -0700

    test changes and code changes

commit e1a032579129a2cd3aae1e93f46570bd57d3343c
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue May 16 09:28:57 2023 -0700

    small changes

commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu May 11 20:36:59 2023 -0700

    Squashed commit of the following:

    commit c757af5f7c630d532bfee5f6dc45aec603ad8a29
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri May 12 04:39:12 2023 +0200

        KAFKA-14752: Kafka examples improvements - demo changes (#13517)

        KAFKA-14752: Kafka examples improvements - demo changes

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 54a4067f81e1434d956ef797274f7b437fe49ea1
    Author: Kamal Chandraprakash <kchandraprakash@uber.com>
    Date:   Thu May 11 21:49:21 2023 +0530

        KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060)

        Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org>

    commit bd65db82b4bad623b0bb31398979e466978148da
    Author: Josep Prat <josep.prat@aiven.io>
    Date:   Thu May 11 17:55:26 2023 +0200

        MINOR: clean up unused methods in core utils (#13706)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org>

    commit 38adb3956979d7025d148612975cc0b82200b2e1
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Thu May 11 07:51:41 2023 -0400

        MINOR: add test tag for testDeadToDeadIllegalTransition (#13694)

        Reviewers: David Jacot <djacot@confluent.io>

    commit ee4132863553fb4fd2df715b2fbd77f349f978b8
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Thu May 11 11:19:32 2023 +0200

        KAFKA-14752: Kafka examples improvements - processor changes (#13516)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a263627adb75f1ca5c87f1482cc70b994ba49d63
    Author: Mickael Maison <mimaison@users.noreply.github.com>
    Date:   Thu May 11 11:02:45 2023 +0200

        MINOR: Remove unused methods in CoreUtils (#13170)

        Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com>

    commit 920a3601ffa7266edf12f3559cfe97c8a5929d03
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Thu May 11 04:13:29 2023 +0200

        MINOR: fix a small typo in SharedServer.scala (#13693)

        Diabled -> Disabled

    commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc
    Author: A. Sophie Blee-Goldman <sophie@confluent.io>
    Date:   Wed May 10 13:39:15 2023 -0700

        HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695)

        The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen()

        Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io>

    commit f17fb75b2de32512f14cb94a7d1bfb0f37485780
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Wed May 10 16:41:52 2023 +0200

        KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690)

        Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 4653507926a42dccda5c086fcae6278afcfc53ca
    Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
    Date:   Wed May 10 05:09:12 2023 -0700

        KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443)

        This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers.

        Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io>

    commit 625ef176ee5f167786003c2d88498632b0b7014b
    Author: Luke Chen <showuon@gmail.com>
    Date:   Wed May 10 16:40:20 2023 +0800

        MINOR: remove kraft readme link (#13691)

        The config/kraft/README.md is already removed. We should also remove the link.

        Reviewers: dengziming <dengziming1993@gmail.com>

    commit 228434d23583189cdcaa7f4a90ebb178ccc17c73
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue May 9 10:49:27 2023 -0400

        KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644)

        This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java.

        Reviewers: David Jacot <djacot@confluent.io>

    commit 59ba9dbbc927ddc8660d0d98d9422909fd306758
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 9 17:58:45 2023 +0530

        KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688)

        `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods.

        This refactoring changes no functionality other than restoring the older methods.

        Reviewers: Randall Hauch <rhauch@gmail.com>

    commit b40a7fc037bb1543c3355fad9c71570f770f5177
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon May 8 14:24:11 2023 -0700

        HOTFIX: fix broken Streams upgrade system test (#13654)

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io>

    commit 7634eee2627da39937e3112ffc58bd7cfedc98f2
    Author: David Jacot <djacot@confluent.io>
    Date:   Mon May 8 20:46:07 2023 +0200

        KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638)

        This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 86daf8ce6573eb79d6e78381dbab738055f914c4
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Mon May 8 20:09:47 2023 +0530

        KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594)

        #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR.

        Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places.

        ---------

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Mon May 8 15:24:52 2023 +0100

        KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621)

        Migrates ChangeLogReader mock in TaskManagerTest to mockito.

        Reviewer: Bruno Cadonna <cadonna@apache.org>

    commit 347238948b86882a47faee4a2916d1b01333d95f
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Mon May 8 07:36:36 2023 +0100

        KAFKA-14662: Update the ACL list in the doc (#13660)

        Added the missing ACLs to the doc.

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a556def5724efb1dc96bd2d389411a8d2f802f53
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Mon May 8 08:35:14 2023 +0200

        MINOR: Print the cause of failure for test (#13672)

        Motivation

        PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build):

        org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
        	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
        	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228)
        	at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is.
        Change

            1. Make the test a bit more robust by waiting for server startup.
            2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure.

        Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

    commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 08:33:31 2023 +0200

        MINOR: Fix producer Callback comment (#13669)

        Fixes the wrong exception name: OffsetMetadataTooLargeException.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 04:15:52 2023 +0200

        KAFKA-14752: Kafka examples improvements - producer changes (#13515)

        KAFKA-14752: Kafka examples improvements - producer changes

        Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com>

    commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Sat May 6 02:56:26 2023 +0800

        MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659)

        Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro>

    commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 14:05:20 2023 +0200

        KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312)

        Motivation

        Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit.

        In this PR, we only update varInt32 encoding/decoding.
        Changes

        This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation.
        Results

        Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java

        Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com>

    commit e34f88403159cc8381da23dafdf7e3d7403114a2
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 13:55:17 2023 +0200

        KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623)

        When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io>

    commit 0822ce0ed1a106a510930bc9ac53a266f54684d7
    Author: David Arthur <mumrah@gmail.com>
    Date:   Fri May 5 04:35:26 2023 -0400

        KAFKA-14840: Support for snapshots during ZK migration (#13461)

        This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active
        controller loaded a snapshot, it would get out of sync with the ZK state.

        In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to
        compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods
        from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and
        AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass
        and without using too much memory.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com>

    commit 97c36f3f3142580325daa1a6aadb662893390561
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 12:20:33 2023 -0700

        HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540

    commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 11:18:03 2023 -0700

        MINOR: improve QuorumController logging #13540

        When creating the QuorumController, log whether ZK migration is enabled.

        When applying a feature level record which sets the metadata version, log the metadata version enum
        rather than the numeric feature level.

        Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the
        end of replay.

        When TRACE is enabled, log every record that is replayed in QuorumController. Since some records
        may contain sensitive information, create RecordRedactor to assist in logging only what is safe to
        put in the log4j file.

        Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged
        at INFO, and additions are logged at TRACE.

        Remove SnapshotReason.java, SnapshotReasonTest.java, and
        QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to
        org.apache.kafka.image.publisher.SnapshotGenerator.

        Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu May 4 09:55:45 2023 -0700

        KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607)

        Also modifies verification to only add a partition to verify if it is transactional.

        When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different.

        Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit ea81e99e5980c807414651034a8c60426a158ca4
    Author: Philip Nee <pnee@confluent.io>
    Date:   Thu May 4 09:20:01 2023 -0700

        KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149)

        Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer.

        This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin.

        Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest.

        Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dc7819d7f1fe6b0160cd95246420ab10c335410b
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Thu May 4 11:00:33 2023 +0100

        KAFKA-14594: Move LogDirsCommand to tools module (#13122)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d46c3f259cce25c43f20fba3943d5cb34ed909ea
    Author: David Mao <47232755+splett2@users.noreply.github.com>
    Date:   Wed May 3 17:09:43 2023 -0700

        MINOR: Reduce number of threads created for integration test brokers (#13655)

        The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker.

        Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io>

    commit c08120f83f7318f15dcf14d525876d18caf6afd0
    Author: Jason Gustafson <jason@confluent.io>
    Date:   Wed May 3 15:25:32 2023 -0700

        MINOR: Allow tagged fields with version subset of flexible version range (#13551)

        The generated message types are missing a range check for the case when the tagged version range is a subset of
        the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the
        number of tags serialized.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b620c03ccf48d6d92b219cba35bb1e5e248d2547
    Author: Luke Chen <showuon@gmail.com>
    Date:   Thu May 4 01:08:25 2023 +0800

        KAFKA-14946: fix NPE when merging the deltatable (#13653)

        Fix NPE while merging the deltatable. Because it's possible that hashTier is
        not null but deltatable is null (ex: removing data), we should have null check
        while merging for deltatable like other places did. Also added tests that will
        fail without this change.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 4a0b6ebf60ed7614f042443460b490971e8662a4
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 2 23:16:46 2023 +0530

        KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657)

        Reviewers: Chris Egerton <chrise@aiven.io>

    commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue May 2 18:04:50 2023 +0200

        KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637)

        This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Tue May 2 16:39:31 2023 +0100

        MINOR: Remove unnecessary code in client/connect (#13259)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Tue May 2 14:28:42 2023 +0200

        KAFKA-14752: Kafka examples improvements - consumer changes (#13514)

        KAFKA-14752: Kafka examples improvements - consumer changes

        This is extracted from the original PR for better review.
        https://github.com/apache/kafka/pull/13492

        Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

        Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 141c76a2c904705f2cd484e96767fcb217c5db25
    Author: Bruno Cadonna <cadonna@apache.org>
    Date:   Tue May 2 14:00:34 2023 +0200

        KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529)

        1. Migrates topology builder mock in TaskManagerTest to mockito.

        2. Replaces the unit test to verify if subscribed partitions are added
        to topology metadata.

        3. Modifies signatures of methods for adding subscribed partitions to
        topology metadata to use sets instead of lists. This makes the
        intent of the methods clearer and makes the tests more portable.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org>

    commit 21af1918eafa30812a955c3c0295b9e968841cd3
    Author: Luke Chen <showuon@gmail.com>
    Date:   Tue May 2 09:54:12 2023 +0800

        MINOR: Add reason to exceptions in QuorumController (#13648)

        Saw this error message in log:

        ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController)

        The null reason is not helpful with only RejectedExecutionException. Adding the reason to it.

        Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit 4773961a44d1f0d1e11d662c7e0fc955027bced2
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon May 1 21:51:09 2023 +0530

        MINOR: Fix Javadoc for configureAdminResources  in Connect's RestServer (#13635)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io>

    commit e29942347acc70aa85d47e84e2021f9c24cd7c80
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Mon May 1 09:56:04 2023 -0400

        KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628)

        Handle migrating SCRAM records in ZK when migrating from ZK to KRaft.

        This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes.

        Reviewers: David Arthur <mumrah@gmail.com>

    commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 08:46:40 2023 -0700

        MINOR: Fixing typos in the ConsumerCoordinator (#13618)

        Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io>

    commit e55fbceb6667cc1455f7fb2e96421c85741fa7df
    Author: Anton Agestam <anton.agestam@aiven.io>
    Date:   Fri Apr 28 11:54:28 2023 +0100

        MINOR: Fix incorrect description of SessionLifetimeMs (#13649)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 02:08:32 2023 -0700

        KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550)

        This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS.

        This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method.  In short, it should no long check and validate only the highest generation.  Instead, we consider 3 cases:
        1. Member will continue to hold on to its partition if there are no other owners
        2. If there are 1+ owners to the same partition. One with the highest generation will win.
        3. If two members of the same generation hold on to the same partition.  We will log an error but remove both from the assignment. (Same with the current logic)

        Here are some important notes that lead to the patch:
        - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown.
        - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error.  This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive

        To summarize the current behavior of different errors:
        `REBALANCE_IN_PROGRESS`
        - heartbeat: requestRejoin if member state is stable
        - joinGroup: rejoin immediately
        - syncGroup: rejoin immediately
        - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress.

        `UNKNOWN_MEMBER_ID`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately
        - syncGroup:  resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        `ILLEGAL_GENERATION`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing.  then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        Reviewers: David Jacot <djacot@confluent.io>

    commit 10b3e667132934084a2d275a204a1a782c2df94e
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri Apr 28 10:32:11 2023 +0200

        KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d796480fe87fd819fc0ac560ca318759180d4644
    Author: Luke Chen <showuon@gmail.com>
    Date:   Fri Apr 28 14:35:12 2023 +0800

        KAFKA-14909: check zkMigrationReady tag before migration (#13631)

        1. add ZkMigrationReady in apiVersionsResponse
        2. check all nodes if ZkMigrationReady are ready before moving to next migration state

        Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com>

    commit c708f7ba5f4f449920cec57a5b69e84e92128b54
    Author: Colin Patrick McCabe <cmccabe@apache.org>
    Date:   Thu Apr 27 19:15:26 2023 -0700

        MINOR: remove spurious call to fatalFaultHandler (#13651)

        Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805.  We should only
        invoke the fatal fault handller if we are unable to generate the activation records. If we are
        unable to write the activation records, a controller failover should be sufficient to remedy the
        situation.

        Co-authored-by: Luke Chen showuon@gmail.com

        Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com>

    commit 056657d84d84e116ffc9460872945b4d2b479ff3
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 17:04:56 2023 -0400

        MINOR add license to reviewers.py

    commit a08f31ecfefca1a51d64137a344e62a236740e62
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 14:32:19 2023 -0400

        MINOR: Adding reviewers.py (#11096)

        This script can be used to help build the "Reviewers: " string we include in commit messages.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit 70493336171363cfa95237a0fe14ef57090553e4
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Wed Apr 26 16:10:46 2023 -0700

        KAFKA-14943: Fix ClientQuotaControlManager validation

        Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be
        used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of
        internal ZK representation details that treated them both as "client configs.")

        Add unit tests for ClientQuotaControlManager.isValidIpEntity and
        ClientQuotaControlManager.configKeysForEntityType.

        This change doesn't affect metadata record application, only input validation. If there are bad
        client quotas that are set currently, this change will not alter the current behavior (of throwing
        an exception and ignoring the bad quota).

    commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051
    Author: David Jacot <djacot@confluent.io>
    Date:   Thu Apr 27 14:05:41 2023 +0200

        KAFKA-14462; [9/N] Add RecordHelpers (#13544)

        This patch adds RecordHelpers.

        Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9
    Author: LinShunKang <linshunkang.chn@gmail.com>
    Date:   Thu Apr 27 10:44:08 2023 +0800

        KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545)

        This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
        Direct use ByteBuffer instead of byte[] to deserialize.

        Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro>

    commit c1b5c75d9271638776392822a094e9e7ef37f490
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 26 10:20:30 2023 -0400

        KAFKA-14805 KRaft controller supports pre-migration mode (#13407)

        This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode,
        the controller will only allow certain write operations. The purpose of this is to disallow metadata
        changes when the controller is waiting for the ZK migration records to be committed.

        The following ControllerWriteEvent operations are permitted in pre-migration mode

        * completeActivation
        * maybeFenceReplicas
        * writeNoOpRecord
        * processBrokerHeartbeat
        * registerBroker (only for migrating ZK brokers)
        * unregisterBroker

        Raft events and other controller events do not follow the same code path as ControllerWriteEvent,
        so they are not affected by this new behavior.

        This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState

        In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean
        MIGRATION rather than PRE_MIGRATION.

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 007c0d375a6e70aefb65f58f9f096016c4cbea16
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 26 17:53:56 2023 +0530

        KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634)

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit baf127a6633161cb52747467880b006d2f54d3bd
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Wed Apr 26 00:30:13 2023 -0700

        KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429)

        Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io>

    commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Tue Apr 25 22:39:23 2023 -0400

        KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit a7d0b3f753708a93aea92e614833f6f6e7443234
    Author: Said Boudjelda <bmscomp@gmail.com>
    Date:   Tue Apr 25 23:31:04 2023 +0200

        MINOR: Upgrade gradle to 8.1.1 (#13625)

        Also upgrade gradle plugins:
         - `org.owasp.dependencycheck` gradle plugin to version `8.2.1`
         - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1`

        Gradle release notes:
        * https://docs.gradle.org/8.1.1/release-notes.html

        Reviewers: Ismael Juma <ismael@juma.me.uk>

    commit 9a36da12b7359b7158332c541655716312efb5b3
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue Apr 25 18:50:51 2023 +0200

        KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538)

        This patch adds ConsumerGroupMember.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 4780dc773f2cd5a20fa5be38d20137745690a888
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue Apr 25 21:20:35 2023 +0530

        KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Tue Apr 25 13:28:37 2023 +0100

        KAFKA-14592: Move FeatureCommand to tools (#13459)

        KAFKA-14592: Move FeatureCommand to tools

        Reviewers: Luke Chen <showuon@gmail.com>

    commit d83a734c41c43a03aa571e602cf8535f0893fd79
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 25 12:02:24 2023 +0200

        MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626)

        The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation
        in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call.

        Reviewers:   maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com>

    commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 15:29:57 2023 -0700

        KAFKA-12446: update change encoding to use varint (#13533)

        KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>,  Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io>

    commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Mon Apr 24 17:06:26 2023 -0400

        KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit 6dcdb017327587a6943fa868595fa3488c7f7ef7
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 12:40:25 2023 -0700

        KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592)

        Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks.

        This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io>

    commit 2271e748a11919d07698ebce759dca2e3075596a
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Mon Apr 24 17:21:19 2023 +0800

        MINOR: fix zookeeper_migration_test.py (#13620)

        Reviewers: Mickael Maison <mimaison@users.noreply.github.com>

    commit c3241236296f9aaee103e475f242e32f04d4c256
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon Apr 24 14:06:20 2023 +0530

        KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 7061475445cb6314e7cf4f9848384224b14f4395
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Fri Apr 21 12:55:41 2023 -0700

        KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575)

        Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest`

        Reviewers: Josep Prat <jlprat@apache.org>

    commit ecdef88f744410039b88e90a5979078d5735aa06
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Fri Apr 21 12:48:05 2023 -0700

        MINOR: updated KS release notes for 3.5 (#13577)

        Reviewers: Walker Carlson <wcarlson@confluent.io>

    commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Fri Apr 21 15:02:06 2023 +0200

        MINOR: fix noticed typo in raft and metadata projects (#13612)

        Reviewers: Josep Prat <jlprat@apache.org>

    commit c39bf714bbaf2a632be4c2a7e446553fe40ba129
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:22:16 2023 +0200

        KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537)

        This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 2d0b816150c79057c813387bd126523b6326a1fc
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:19:04 2023 +0200

        MINOR: Move `ControllerPurgatory` to `server-common` (#13555)

        This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit df137752542c005c6998c37c03222ffbeca0f349
    Author: Purshotam Chauhan <pchauhan@confluent.io>
    Date:   Fri Apr 21 14:08:23 2023 +0530

        KAFKA-14828: Remove R/W locks using persistent data structures (#13437)

        Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only.

        In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation.

        Reviewers:  Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>,  Divij Vaidya <diviv@amazon.com>

    commit 2ee770ac7e576c85a911ccb307339be3d58a8942
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu Apr 20 10:21:26 2023 -0700

        Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)"

        This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc.

    commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu Apr 20 09:30:11 2023 -0700

        KAFKA-14904: Pending state blocked verification of transactions (#13579)

        KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction.

        There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes.

        When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions.

        The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely.

        This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 7f175feacaba4187bbb3631dff3a1330060f191e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Thu Apr 20 13:50:29 2023 +0530

        KAFKA-14586: Adding redirection for StreamsResetter (#13614)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

    commit e14dd8024adae746c50c8b7d9cd268e859669576
    Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com>
    Date:   Thu Apr 20 05:29:27 2023 +0200

        KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432)

        We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic:
        1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry.
        2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of  the client.
        3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`.

        We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them.

        Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io>

    commit f5de4daa71f0ad31aa64443c5ff43c59712feea5
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Wed Apr 19 19:45:26 2023 -0400

        KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires

        FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change
        events. However, it does not distinguish between issues related to feature changes actually failing and other
        exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration
        could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between
        these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues.
        We shutdown the broker only for FeatureCacheUpdateException.

        Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 11c8bf4826197533807b2132cfc6599ba70de1c1
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Wed Apr 19 19:34:36 2023 -0400

        KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609)

        This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter.

        Part of KIP-914.

        Reviewer: Matthias J. Sax <matthias@confluent.io>

    commit 809966a9a06664e0b521c2298fa0de834e443607
    Author: Matthew de Detrich <matthew.dedetrich@aiven.io>
    Date:   Wed Apr 19 20:54:07 2023 +0200

        KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478)

        Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address.

        Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org>

    commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 19 14:19:13 2023 -0400

        KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606)

        This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b10716e72370c4e128bddb17bcc107ccab221e47
    Author: hudeqi <1217150961@qq.com>
    Date:   Thu Apr 20 00:49:08 2023 +0800

        KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com>

    commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc
    Author: Keith Wall <kwall@redhat.com>
    Date:   Wed Apr 19 03:58:29 2023 +0100

        KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)

        Changes SocketServer to set the setReuseAddress(true) socket option.

        This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test.

        Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com>

    commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 19 00:46:03 2023 +0530

        MINOR: Fixing gradle build during compileScala and compileTestScala (#13588)

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

    commit 3388adf1b52f545e2d69edd30cdd53241b2887f3
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Tue Apr 18 11:32:27 2023 -0700

        MINOR: rename internal FK-join processor classes (#13589)

        Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>

    commit abca86511ecc9e081d676976ff6d3b845308f444
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Tue Apr 18 12:41:38 2023 -0400

        KAFKA-14881: Rework UserScramCredentialRecord (#13513)

        Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This
        is necessary to support migration from ZK, since those are the fields we stored in ZK.  Update
        latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version.  Moved
        ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more
        appropriate.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 61530d68ce83467de6190a52da37b3c0af84f0ef
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 09:37:04 2023 -0400

        KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526)

        This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields.

        Reviewers: David Jacot <djacot@confluent.io>

    commit b36a170aa3b2738177d7229859db765e0115385b
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 18 13:36:56 2023 +0200

        MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 5e9d4de748dff7b91043a9b799716ab4becdae7e
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 04:41:54 2023 -0400

        KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511)

        This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io>

    commit 454b72161a76b1687a1263157d7cc30a1bdb2506
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Tue Apr 18 09:40:14 2023 +0200

        KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557)

        KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow.
        To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread.

    commit e27926f92b1f6b34ed6731f33c712a5d0d594275
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Mon Apr 17 17:52:28 2023 -0400

        KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280)

        topic counts.

        Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change.  Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events.  We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods.

        Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>

    commit 7159f6c1a81ead277030f55f293943270346ad4e
    Author: Alyssa Huang <ahuang@confluent.io>
    Date:   Mon Apr 17 11:03:45 2023 -0700

        MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields

        Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it
        should have been. Add a regression test.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit cc88822986e4004e772f376eae47dbe08e29d137
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 10 16:29:27 2023 -0700

    small changes

commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 25 14:58:20 2023 -0700

    grammar and code comments changes

commit 5550e4934ade28f069e4ae9ab97e76982ff3401f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 17:39:48 2023 -0700

    fixed formatting stuff

commit 28ab29819bf327309ce07196a3fb0492f7fb2281
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 11:38:57 2023 -0700

    consumers -> members

commit e56ded267456bbe1c22ded0d037953545b921d1d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 11:26:23 2023 -0700

    Interface changes incorporated

commit 2d50c278c18baa11b0235330149dd8f9def57c3d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 13 14:47:33 2023 -0700

    Renamed topicIdToPartition class

commit d33bbfac053ccac77204ac4253698ed356d7c621
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 17:31:26 2023 -0700

    Added stickiness test

commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 13:28:24 2023 -0700

    Separated uniform and general builder in different files, added more tests

commit ef814867aba68fc4813359c5cef6ae155ed49a60
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:24:27 2023 -0700

    import changes

commit dffad677c6e4a33625401d621f046af6f21feb99
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:16:33 2023 -0700

    Fixed checkstyle and added test file

commit 7fd779418df3504a490c0d10f3848360a4417077
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 5 15:13:38 2023 -0700

    First draft optimised uniform assignor code

commit 309cc6141f7962e794c90d4107f460fafd51697d
Merge: 3d9264547c 2c1cf03a89
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Mon Apr 17 11:03:07 2023 -0700

    Merge branch 'trunk' into rreddy-22/KAFKA-14514

commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 10:58:16 2023 -0700

    Interface changes incorporated, added stickiness test and non-existent topic test.

commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436
Merge: a1bdb58e08 57b94ca208
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 09:49:23 2023 -0700

    Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514

    # Conflicts:
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java

commit a1bdb58e082d98b73f50860db5b35684f4eba51b
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 20:01:59 2023 -0700

    Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc.

commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 11:28:40 2023 -0700

    Addressed some PR comments

commit 918d262ae52892c13766649db847ede6928921f8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7369db3744b4c89f625f6b73db331eaf0512c4eb
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit a925b02dd3d78c8b9c1415c083f0544edcb404fe
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit 48a982232c6f1ff371cb44333fd12d6b37ef381a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation

commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 13:00:07 2023 -0700

    Removed topicIdToPartition class, changed attribute names and added java doc for getter methods

commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 15:09:44 2023 -0700

    Made all attributes private and added getter methods, changed names according to PR comments

commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:25:33 2023 -0700

    Added toString method and hash

commit 5065109332845541c5591a52683f21d37982d80a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 13:43:26 2023 -0700

    Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure

commit 57b94ca208cb374b22d61eb044ef8004ea09479f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7352cbb5683f676382f56e6ae5c926f3cae98824
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0
Merge: 4b321aa537 637bc92ba1
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Thu Apr 6 10:37:58 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514

commit 4b321aa5374427a15346be9b87b7c545ccb7db61
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 74105291cc687fe41b75de692fc624ab1335341a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit 3df34605e078beb9babcbeadc7b119a70da63796
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation
rreddy-22 added a commit to rreddy-22/kafka-rreddy that referenced this pull request Jul 24, 2023
@rreddy-22
Squashed commit of the following:
8530788 
commit 1c955b5e251aedc174cf28f43037f08162d89b49
Merge: 2d40125fa8 81f1ccd7a2
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 13:02:54 2023 -0700

    Merge branch 'rreddy-22/Rack-Assignor-Interface-Changes' of github.com:rreddy-22/kafka-rreddy into rreddy-22/Rack-Assignor-Interface-Changes

commit 81f1ccd7a25d85eaf1cff713df402e0100d4b4a3
Merge: a39e76f63f 84691b11f6
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Mon Jul 24 12:56:40 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes

commit 2d40125fa8ba16b4f07e6bfe410ceacfd8a112e8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 11:45:21 2023 -0700

    minor

commit a39e76f63f312ecacdcebd2c5015777e160ca0b3
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 10:04:10 2023 -0700

    reverted reviewers.py changes

commit c0b464c8936f1df4f6540b8abe7c1fa88fce6e81
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 00:02:03 2023 -0700

    reverted grammar changes

commit 6fe72dfce921ab9d77ba6b28d4104c5d92c6dbf3
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 23:56:35 2023 -0700

    minor

commit a30a53463e55b72699617360d1c888b7ee310336
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 23:55:35 2023 -0700

    minor

commit 57b9a6233a4f697f285b85e6394d3dd407f16404
Merge: 061dac797b 4981fa939d
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Sun Jul 23 19:52:07 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes

commit 061dac797bae56d91659fbc8d7900117eb10a866
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:43:12 2023 -0700

    Moved SubscribedTopicMetadata to the consumer package

commit 6650ea5cd707f03713fc2c69ba76fafd56d886b8
Merge: 22f2df02de faafed25a1
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:33:17 2023 -0700

    Merge remote-tracking branch 'origin/rreddy-22/Rack-Assignor-Interface-Changes' into rreddy-22/Rack-Assignor-Interface-Changes

commit 22f2df02deabe1a6b5b8b8cceb432486b9ef4470
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:25:11 2023 -0700

    minor changes

commit faafed25a10d7ff79fb8a24ab62aa222c0e6516a
Merge: 4511683d4e 1bf73d89d0
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Fri Jul 21 12:50:59 2023 -0700

    Merge branch 'trunk' into rreddy-22/Rack-Assignor-Interface-Changes

commit 4511683d4ef489cb979769ff6775dfe0250cdeeb
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 21 03:09:27 2023 -0700

    Changes based on PR comments

commit 579bb948fcc78afd163410a5bc43832bc87aaf71
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Jul 19 14:45:18 2023 -0700

    New interfaces and classes added to facilitate passing rack information to the assignor, modified tests to incorporate changes

commit f68471a99f2e0260b36047a7596b0f9de48794ad
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Jul 13 11:55:20 2023 -0700

    Small edits

commit c50f3bdbf79f1ca127355d2a9002b1fdbf0baa2f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 11 16:30:32 2023 -0700

    Removed extra lines

commit 478488e0943aa879b27eec5ae5eb79a23dd9cab3
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 11 16:21:54 2023 -0700

    Added rack interface changes, added a new TopicAndClusterMetadata class to handle topic and cluster images and added an AbstractPartitionAssignor

commit ce540f943b8dae0786a87e51cafacb66efd1e64d
Merge: 9815d6db04 fd5b300b57
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:45:27 2023 -0600

    Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized

commit 9815d6db04c264456b7ead7fc5f735513a868c7d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:23:00 2023 -0600

    Abstract Partition Assignor

commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Jun 29 16:36:53 2023 -0700

    Refactored code to reduce format conversion time

commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 31 16:21:01 2023 -0700

    test changes and code changes

commit e1a032579129a2cd3aae1e93f46570bd57d3343c
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue May 16 09:28:57 2023 -0700

    small changes

commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu May 11 20:36:59 2023 -0700

    Squashed commit of the following:

    commit c757af5f7c630d532bfee5f6dc45aec603ad8a29
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri May 12 04:39:12 2023 +0200

        KAFKA-14752: Kafka examples improvements - demo changes (#13517)

        KAFKA-14752: Kafka examples improvements - demo changes

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 54a4067f81e1434d956ef797274f7b437fe49ea1
    Author: Kamal Chandraprakash <kchandraprakash@uber.com>
    Date:   Thu May 11 21:49:21 2023 +0530

        KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060)

        Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org>

    commit bd65db82b4bad623b0bb31398979e466978148da
    Author: Josep Prat <josep.prat@aiven.io>
    Date:   Thu May 11 17:55:26 2023 +0200

        MINOR: clean up unused methods in core utils (#13706)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org>

    commit 38adb3956979d7025d148612975cc0b82200b2e1
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Thu May 11 07:51:41 2023 -0400

        MINOR: add test tag for testDeadToDeadIllegalTransition (#13694)

        Reviewers: David Jacot <djacot@confluent.io>

    commit ee4132863553fb4fd2df715b2fbd77f349f978b8
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Thu May 11 11:19:32 2023 +0200

        KAFKA-14752: Kafka examples improvements - processor changes (#13516)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a263627adb75f1ca5c87f1482cc70b994ba49d63
    Author: Mickael Maison <mimaison@users.noreply.github.com>
    Date:   Thu May 11 11:02:45 2023 +0200

        MINOR: Remove unused methods in CoreUtils (#13170)

        Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com>

    commit 920a3601ffa7266edf12f3559cfe97c8a5929d03
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Thu May 11 04:13:29 2023 +0200

        MINOR: fix a small typo in SharedServer.scala (#13693)

        Diabled -> Disabled

    commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc
    Author: A. Sophie Blee-Goldman <sophie@confluent.io>
    Date:   Wed May 10 13:39:15 2023 -0700

        HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695)

        The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen()

        Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io>

    commit f17fb75b2de32512f14cb94a7d1bfb0f37485780
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Wed May 10 16:41:52 2023 +0200

        KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690)

        Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 4653507926a42dccda5c086fcae6278afcfc53ca
    Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
    Date:   Wed May 10 05:09:12 2023 -0700

        KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443)

        This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers.

        Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io>

    commit 625ef176ee5f167786003c2d88498632b0b7014b
    Author: Luke Chen <showuon@gmail.com>
    Date:   Wed May 10 16:40:20 2023 +0800

        MINOR: remove kraft readme link (#13691)

        The config/kraft/README.md is already removed. We should also remove the link.

        Reviewers: dengziming <dengziming1993@gmail.com>

    commit 228434d23583189cdcaa7f4a90ebb178ccc17c73
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue May 9 10:49:27 2023 -0400

        KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644)

        This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java.

        Reviewers: David Jacot <djacot@confluent.io>

    commit 59ba9dbbc927ddc8660d0d98d9422909fd306758
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 9 17:58:45 2023 +0530

        KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688)

        `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods.

        This refactoring changes no functionality other than restoring the older methods.

        Reviewers: Randall Hauch <rhauch@gmail.com>

    commit b40a7fc037bb1543c3355fad9c71570f770f5177
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon May 8 14:24:11 2023 -0700

        HOTFIX: fix broken Streams upgrade system test (#13654)

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io>

    commit 7634eee2627da39937e3112ffc58bd7cfedc98f2
    Author: David Jacot <djacot@confluent.io>
    Date:   Mon May 8 20:46:07 2023 +0200

        KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638)

        This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 86daf8ce6573eb79d6e78381dbab738055f914c4
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Mon May 8 20:09:47 2023 +0530

        KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594)

        #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR.

        Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places.

        ---------

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Mon May 8 15:24:52 2023 +0100

        KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621)

        Migrates ChangeLogReader mock in TaskManagerTest to mockito.

        Reviewer: Bruno Cadonna <cadonna@apache.org>

    commit 347238948b86882a47faee4a2916d1b01333d95f
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Mon May 8 07:36:36 2023 +0100

        KAFKA-14662: Update the ACL list in the doc (#13660)

        Added the missing ACLs to the doc.

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a556def5724efb1dc96bd2d389411a8d2f802f53
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Mon May 8 08:35:14 2023 +0200

        MINOR: Print the cause of failure for test (#13672)

        Motivation

        PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build):

        org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
        	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
        	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228)
        	at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is.
        Change

            1. Make the test a bit more robust by waiting for server startup.
            2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure.

        Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

    commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 08:33:31 2023 +0200

        MINOR: Fix producer Callback comment (#13669)

        Fixes the wrong exception name: OffsetMetadataTooLargeException.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 04:15:52 2023 +0200

        KAFKA-14752: Kafka examples improvements - producer changes (#13515)

        KAFKA-14752: Kafka examples improvements - producer changes

        Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com>

    commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Sat May 6 02:56:26 2023 +0800

        MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659)

        Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro>

    commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 14:05:20 2023 +0200

        KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312)

        Motivation

        Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit.

        In this PR, we only update varInt32 encoding/decoding.
        Changes

        This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation.
        Results

        Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java

        Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com>

    commit e34f88403159cc8381da23dafdf7e3d7403114a2
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 13:55:17 2023 +0200

        KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623)

        When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io>

    commit 0822ce0ed1a106a510930bc9ac53a266f54684d7
    Author: David Arthur <mumrah@gmail.com>
    Date:   Fri May 5 04:35:26 2023 -0400

        KAFKA-14840: Support for snapshots during ZK migration (#13461)

        This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active
        controller loaded a snapshot, it would get out of sync with the ZK state.

        In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to
        compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods
        from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and
        AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass
        and without using too much memory.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com>

    commit 97c36f3f3142580325daa1a6aadb662893390561
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 12:20:33 2023 -0700

        HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540

    commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 11:18:03 2023 -0700

        MINOR: improve QuorumController logging #13540

        When creating the QuorumController, log whether ZK migration is enabled.

        When applying a feature level record which sets the metadata version, log the metadata version enum
        rather than the numeric feature level.

        Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the
        end of replay.

        When TRACE is enabled, log every record that is replayed in QuorumController. Since some records
        may contain sensitive information, create RecordRedactor to assist in logging only what is safe to
        put in the log4j file.

        Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged
        at INFO, and additions are logged at TRACE.

        Remove SnapshotReason.java, SnapshotReasonTest.java, and
        QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to
        org.apache.kafka.image.publisher.SnapshotGenerator.

        Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu May 4 09:55:45 2023 -0700

        KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607)

        Also modifies verification to only add a partition to verify if it is transactional.

        When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different.

        Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit ea81e99e5980c807414651034a8c60426a158ca4
    Author: Philip Nee <pnee@confluent.io>
    Date:   Thu May 4 09:20:01 2023 -0700

        KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149)

        Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer.

        This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin.

        Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest.

        Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dc7819d7f1fe6b0160cd95246420ab10c335410b
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Thu May 4 11:00:33 2023 +0100

        KAFKA-14594: Move LogDirsCommand to tools module (#13122)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d46c3f259cce25c43f20fba3943d5cb34ed909ea
    Author: David Mao <47232755+splett2@users.noreply.github.com>
    Date:   Wed May 3 17:09:43 2023 -0700

        MINOR: Reduce number of threads created for integration test brokers (#13655)

        The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker.

        Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io>

    commit c08120f83f7318f15dcf14d525876d18caf6afd0
    Author: Jason Gustafson <jason@confluent.io>
    Date:   Wed May 3 15:25:32 2023 -0700

        MINOR: Allow tagged fields with version subset of flexible version range (#13551)

        The generated message types are missing a range check for the case when the tagged version range is a subset of
        the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the
        number of tags serialized.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b620c03ccf48d6d92b219cba35bb1e5e248d2547
    Author: Luke Chen <showuon@gmail.com>
    Date:   Thu May 4 01:08:25 2023 +0800

        KAFKA-14946: fix NPE when merging the deltatable (#13653)

        Fix NPE while merging the deltatable. Because it's possible that hashTier is
        not null but deltatable is null (ex: removing data), we should have null check
        while merging for deltatable like other places did. Also added tests that will
        fail without this change.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 4a0b6ebf60ed7614f042443460b490971e8662a4
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 2 23:16:46 2023 +0530

        KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657)

        Reviewers: Chris Egerton <chrise@aiven.io>

    commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue May 2 18:04:50 2023 +0200

        KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637)

        This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Tue May 2 16:39:31 2023 +0100

        MINOR: Remove unnecessary code in client/connect (#13259)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Tue May 2 14:28:42 2023 +0200

        KAFKA-14752: Kafka examples improvements - consumer changes (#13514)

        KAFKA-14752: Kafka examples improvements - consumer changes

        This is extracted from the original PR for better review.
        https://github.com/apache/kafka/pull/13492

        Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

        Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 141c76a2c904705f2cd484e96767fcb217c5db25
    Author: Bruno Cadonna <cadonna@apache.org>
    Date:   Tue May 2 14:00:34 2023 +0200

        KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529)

        1. Migrates topology builder mock in TaskManagerTest to mockito.

        2. Replaces the unit test to verify if subscribed partitions are added
        to topology metadata.

        3. Modifies signatures of methods for adding subscribed partitions to
        topology metadata to use sets instead of lists. This makes the
        intent of the methods clearer and makes the tests more portable.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org>

    commit 21af1918eafa30812a955c3c0295b9e968841cd3
    Author: Luke Chen <showuon@gmail.com>
    Date:   Tue May 2 09:54:12 2023 +0800

        MINOR: Add reason to exceptions in QuorumController (#13648)

        Saw this error message in log:

        ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController)

        The null reason is not helpful with only RejectedExecutionException. Adding the reason to it.

        Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit 4773961a44d1f0d1e11d662c7e0fc955027bced2
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon May 1 21:51:09 2023 +0530

        MINOR: Fix Javadoc for configureAdminResources  in Connect's RestServer (#13635)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io>

    commit e29942347acc70aa85d47e84e2021f9c24cd7c80
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Mon May 1 09:56:04 2023 -0400

        KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628)

        Handle migrating SCRAM records in ZK when migrating from ZK to KRaft.

        This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes.

        Reviewers: David Arthur <mumrah@gmail.com>

    commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 08:46:40 2023 -0700

        MINOR: Fixing typos in the ConsumerCoordinator (#13618)

        Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io>

    commit e55fbceb6667cc1455f7fb2e96421c85741fa7df
    Author: Anton Agestam <anton.agestam@aiven.io>
    Date:   Fri Apr 28 11:54:28 2023 +0100

        MINOR: Fix incorrect description of SessionLifetimeMs (#13649)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 02:08:32 2023 -0700

        KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550)

        This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS.

        This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method.  In short, it should no long check and validate only the highest generation.  Instead, we consider 3 cases:
        1. Member will continue to hold on to its partition if there are no other owners
        2. If there are 1+ owners to the same partition. One with the highest generation will win.
        3. If two members of the same generation hold on to the same partition.  We will log an error but remove both from the assignment. (Same with the current logic)

        Here are some important notes that lead to the patch:
        - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown.
        - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error.  This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive

        To summarize the current behavior of different errors:
        `REBALANCE_IN_PROGRESS`
        - heartbeat: requestRejoin if member state is stable
        - joinGroup: rejoin immediately
        - syncGroup: rejoin immediately
        - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress.

        `UNKNOWN_MEMBER_ID`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately
        - syncGroup:  resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        `ILLEGAL_GENERATION`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing.  then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        Reviewers: David Jacot <djacot@confluent.io>

    commit 10b3e667132934084a2d275a204a1a782c2df94e
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri Apr 28 10:32:11 2023 +0200

        KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d796480fe87fd819fc0ac560ca318759180d4644
    Author: Luke Chen <showuon@gmail.com>
    Date:   Fri Apr 28 14:35:12 2023 +0800

        KAFKA-14909: check zkMigrationReady tag before migration (#13631)

        1. add ZkMigrationReady in apiVersionsResponse
        2. check all nodes if ZkMigrationReady are ready before moving to next migration state

        Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com>

    commit c708f7ba5f4f449920cec57a5b69e84e92128b54
    Author: Colin Patrick McCabe <cmccabe@apache.org>
    Date:   Thu Apr 27 19:15:26 2023 -0700

        MINOR: remove spurious call to fatalFaultHandler (#13651)

        Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805.  We should only
        invoke the fatal fault handller if we are unable to generate the activation records. If we are
        unable to write the activation records, a controller failover should be sufficient to remedy the
        situation.

        Co-authored-by: Luke Chen showuon@gmail.com

        Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com>

    commit 056657d84d84e116ffc9460872945b4d2b479ff3
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 17:04:56 2023 -0400

        MINOR add license to reviewers.py

    commit a08f31ecfefca1a51d64137a344e62a236740e62
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 14:32:19 2023 -0400

        MINOR: Adding reviewers.py (#11096)

        This script can be used to help build the "Reviewers: " string we include in commit messages.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit 70493336171363cfa95237a0fe14ef57090553e4
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Wed Apr 26 16:10:46 2023 -0700

        KAFKA-14943: Fix ClientQuotaControlManager validation

        Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be
        used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of
        internal ZK representation details that treated them both as "client configs.")

        Add unit tests for ClientQuotaControlManager.isValidIpEntity and
        ClientQuotaControlManager.configKeysForEntityType.

        This change doesn't affect metadata record application, only input validation. If there are bad
        client quotas that are set currently, this change will not alter the current behavior (of throwing
        an exception and ignoring the bad quota).

    commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051
    Author: David Jacot <djacot@confluent.io>
    Date:   Thu Apr 27 14:05:41 2023 +0200

        KAFKA-14462; [9/N] Add RecordHelpers (#13544)

        This patch adds RecordHelpers.

        Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9
    Author: LinShunKang <linshunkang.chn@gmail.com>
    Date:   Thu Apr 27 10:44:08 2023 +0800

        KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545)

        This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
        Direct use ByteBuffer instead of byte[] to deserialize.

        Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro>

    commit c1b5c75d9271638776392822a094e9e7ef37f490
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 26 10:20:30 2023 -0400

        KAFKA-14805 KRaft controller supports pre-migration mode (#13407)

        This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode,
        the controller will only allow certain write operations. The purpose of this is to disallow metadata
        changes when the controller is waiting for the ZK migration records to be committed.

        The following ControllerWriteEvent operations are permitted in pre-migration mode

        * completeActivation
        * maybeFenceReplicas
        * writeNoOpRecord
        * processBrokerHeartbeat
        * registerBroker (only for migrating ZK brokers)
        * unregisterBroker

        Raft events and other controller events do not follow the same code path as ControllerWriteEvent,
        so they are not affected by this new behavior.

        This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState

        In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean
        MIGRATION rather than PRE_MIGRATION.

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 007c0d375a6e70aefb65f58f9f096016c4cbea16
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 26 17:53:56 2023 +0530

        KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634)

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit baf127a6633161cb52747467880b006d2f54d3bd
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Wed Apr 26 00:30:13 2023 -0700

        KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429)

        Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io>

    commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Tue Apr 25 22:39:23 2023 -0400

        KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit a7d0b3f753708a93aea92e614833f6f6e7443234
    Author: Said Boudjelda <bmscomp@gmail.com>
    Date:   Tue Apr 25 23:31:04 2023 +0200

        MINOR: Upgrade gradle to 8.1.1 (#13625)

        Also upgrade gradle plugins:
         - `org.owasp.dependencycheck` gradle plugin to version `8.2.1`
         - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1`

        Gradle release notes:
        * https://docs.gradle.org/8.1.1/release-notes.html

        Reviewers: Ismael Juma <ismael@juma.me.uk>

    commit 9a36da12b7359b7158332c541655716312efb5b3
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue Apr 25 18:50:51 2023 +0200

        KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538)

        This patch adds ConsumerGroupMember.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 4780dc773f2cd5a20fa5be38d20137745690a888
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue Apr 25 21:20:35 2023 +0530

        KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Tue Apr 25 13:28:37 2023 +0100

        KAFKA-14592: Move FeatureCommand to tools (#13459)

        KAFKA-14592: Move FeatureCommand to tools

        Reviewers: Luke Chen <showuon@gmail.com>

    commit d83a734c41c43a03aa571e602cf8535f0893fd79
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 25 12:02:24 2023 +0200

        MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626)

        The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation
        in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call.

        Reviewers:   maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com>

    commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 15:29:57 2023 -0700

        KAFKA-12446: update change encoding to use varint (#13533)

        KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>,  Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io>

    commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Mon Apr 24 17:06:26 2023 -0400

        KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit 6dcdb017327587a6943fa868595fa3488c7f7ef7
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 12:40:25 2023 -0700

        KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592)

        Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks.

        This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io>

    commit 2271e748a11919d07698ebce759dca2e3075596a
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Mon Apr 24 17:21:19 2023 +0800

        MINOR: fix zookeeper_migration_test.py (#13620)

        Reviewers: Mickael Maison <mimaison@users.noreply.github.com>

    commit c3241236296f9aaee103e475f242e32f04d4c256
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon Apr 24 14:06:20 2023 +0530

        KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 7061475445cb6314e7cf4f9848384224b14f4395
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Fri Apr 21 12:55:41 2023 -0700

        KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575)

        Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest`

        Reviewers: Josep Prat <jlprat@apache.org>

    commit ecdef88f744410039b88e90a5979078d5735aa06
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Fri Apr 21 12:48:05 2023 -0700

        MINOR: updated KS release notes for 3.5 (#13577)

        Reviewers: Walker Carlson <wcarlson@confluent.io>

    commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Fri Apr 21 15:02:06 2023 +0200

        MINOR: fix noticed typo in raft and metadata projects (#13612)

        Reviewers: Josep Prat <jlprat@apache.org>

    commit c39bf714bbaf2a632be4c2a7e446553fe40ba129
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:22:16 2023 +0200

        KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537)

        This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 2d0b816150c79057c813387bd126523b6326a1fc
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:19:04 2023 +0200

        MINOR: Move `ControllerPurgatory` to `server-common` (#13555)

        This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit df137752542c005c6998c37c03222ffbeca0f349
    Author: Purshotam Chauhan <pchauhan@confluent.io>
    Date:   Fri Apr 21 14:08:23 2023 +0530

        KAFKA-14828: Remove R/W locks using persistent data structures (#13437)

        Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only.

        In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation.

        Reviewers:  Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>,  Divij Vaidya <diviv@amazon.com>

    commit 2ee770ac7e576c85a911ccb307339be3d58a8942
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu Apr 20 10:21:26 2023 -0700

        Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)"

        This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc.

    commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu Apr 20 09:30:11 2023 -0700

        KAFKA-14904: Pending state blocked verification of transactions (#13579)

        KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction.

        There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes.

        When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions.

        The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely.

        This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 7f175feacaba4187bbb3631dff3a1330060f191e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Thu Apr 20 13:50:29 2023 +0530

        KAFKA-14586: Adding redirection for StreamsResetter (#13614)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

    commit e14dd8024adae746c50c8b7d9cd268e859669576
    Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com>
    Date:   Thu Apr 20 05:29:27 2023 +0200

        KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432)

        We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic:
        1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry.
        2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of  the client.
        3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`.

        We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them.

        Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io>

    commit f5de4daa71f0ad31aa64443c5ff43c59712feea5
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Wed Apr 19 19:45:26 2023 -0400

        KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires

        FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change
        events. However, it does not distinguish between issues related to feature changes actually failing and other
        exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration
        could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between
        these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues.
        We shutdown the broker only for FeatureCacheUpdateException.

        Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 11c8bf4826197533807b2132cfc6599ba70de1c1
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Wed Apr 19 19:34:36 2023 -0400

        KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609)

        This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter.

        Part of KIP-914.

        Reviewer: Matthias J. Sax <matthias@confluent.io>

    commit 809966a9a06664e0b521c2298fa0de834e443607
    Author: Matthew de Detrich <matthew.dedetrich@aiven.io>
    Date:   Wed Apr 19 20:54:07 2023 +0200

        KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478)

        Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address.

        Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org>

    commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 19 14:19:13 2023 -0400

        KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606)

        This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b10716e72370c4e128bddb17bcc107ccab221e47
    Author: hudeqi <1217150961@qq.com>
    Date:   Thu Apr 20 00:49:08 2023 +0800

        KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com>

    commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc
    Author: Keith Wall <kwall@redhat.com>
    Date:   Wed Apr 19 03:58:29 2023 +0100

        KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)

        Changes SocketServer to set the setReuseAddress(true) socket option.

        This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test.

        Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com>

    commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 19 00:46:03 2023 +0530

        MINOR: Fixing gradle build during compileScala and compileTestScala (#13588)

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

    commit 3388adf1b52f545e2d69edd30cdd53241b2887f3
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Tue Apr 18 11:32:27 2023 -0700

        MINOR: rename internal FK-join processor classes (#13589)

        Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>

    commit abca86511ecc9e081d676976ff6d3b845308f444
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Tue Apr 18 12:41:38 2023 -0400

        KAFKA-14881: Rework UserScramCredentialRecord (#13513)

        Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This
        is necessary to support migration from ZK, since those are the fields we stored in ZK.  Update
        latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version.  Moved
        ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more
        appropriate.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 61530d68ce83467de6190a52da37b3c0af84f0ef
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 09:37:04 2023 -0400

        KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526)

        This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields.

        Reviewers: David Jacot <djacot@confluent.io>

    commit b36a170aa3b2738177d7229859db765e0115385b
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 18 13:36:56 2023 +0200

        MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 5e9d4de748dff7b91043a9b799716ab4becdae7e
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 04:41:54 2023 -0400

        KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511)

        This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io>

    commit 454b72161a76b1687a1263157d7cc30a1bdb2506
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Tue Apr 18 09:40:14 2023 +0200

        KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557)

        KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow.
        To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread.

    commit e27926f92b1f6b34ed6731f33c712a5d0d594275
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Mon Apr 17 17:52:28 2023 -0400

        KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280)

        topic counts.

        Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change.  Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events.  We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods.

        Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>

    commit 7159f6c1a81ead277030f55f293943270346ad4e
    Author: Alyssa Huang <ahuang@confluent.io>
    Date:   Mon Apr 17 11:03:45 2023 -0700

        MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields

        Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it
        should have been. Add a regression test.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit cc88822986e4004e772f376eae47dbe08e29d137
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 10 16:29:27 2023 -0700

    small changes

commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 25 14:58:20 2023 -0700

    grammar and code comments changes

commit 5550e4934ade28f069e4ae9ab97e76982ff3401f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 17:39:48 2023 -0700

    fixed formatting stuff

commit 28ab29819bf327309ce07196a3fb0492f7fb2281
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 11:38:57 2023 -0700

    consumers -> members

commit e56ded267456bbe1c22ded0d037953545b921d1d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 11:26:23 2023 -0700

    Interface changes incorporated

commit 2d50c278c18baa11b0235330149dd8f9def57c3d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 13 14:47:33 2023 -0700

    Renamed topicIdToPartition class

commit d33bbfac053ccac77204ac4253698ed356d7c621
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 17:31:26 2023 -0700

    Added stickiness test

commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 13:28:24 2023 -0700

    Separated uniform and general builder in different files, added more tests

commit ef814867aba68fc4813359c5cef6ae155ed49a60
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:24:27 2023 -0700

    import changes

commit dffad677c6e4a33625401d621f046af6f21feb99
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:16:33 2023 -0700

    Fixed checkstyle and added test file

commit 7fd779418df3504a490c0d10f3848360a4417077
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 5 15:13:38 2023 -0700

    First draft optimised uniform assignor code

commit 309cc6141f7962e794c90d4107f460fafd51697d
Merge: 3d9264547c 2c1cf03a89
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Mon Apr 17 11:03:07 2023 -0700

    Merge branch 'trunk' into rreddy-22/KAFKA-14514

commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 10:58:16 2023 -0700

    Interface changes incorporated, added stickiness test and non-existent topic test.

commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436
Merge: a1bdb58e08 57b94ca208
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 09:49:23 2023 -0700

    Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514

    # Conflicts:
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java

commit a1bdb58e082d98b73f50860db5b35684f4eba51b
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 20:01:59 2023 -0700

    Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc.

commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 11:28:40 2023 -0700

    Addressed some PR comments

commit 918d262ae52892c13766649db847ede6928921f8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7369db3744b4c89f625f6b73db331eaf0512c4eb
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit a925b02dd3d78c8b9c1415c083f0544edcb404fe
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit 48a982232c6f1ff371cb44333fd12d6b37ef381a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation

commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 13:00:07 2023 -0700

    Removed topicIdToPartition class, changed attribute names and added java doc for getter methods

commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 15:09:44 2023 -0700

    Made all attributes private and added getter methods, changed names according to PR comments

commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:25:33 2023 -0700

    Added toString method and hash

commit 5065109332845541c5591a52683f21d37982d80a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 13:43:26 2023 -0700

    Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure

commit 57b94ca208cb374b22d61eb044ef8004ea09479f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7352cbb5683f676382f56e6ae5c926f3cae98824
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0
Merge: 4b321aa537 637bc92ba1
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Thu Apr 6 10:37:58 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514

commit 4b321aa5374427a15346be9b87b7c545ccb7db61
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 74105291cc687fe41b75de692fc624ab1335341a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit 3df34605e078beb9babcbeadc7b119a70da63796
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation
rreddy-22 added a commit to rreddy-22/kafka-rreddy that referenced this pull request Jul 24, 2023
@rreddy-22
Squashed commit of the following:
003215f 
commit ce540f943b8dae0786a87e51cafacb66efd1e64d
Merge: 9815d6db04 fd5b300b57
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:45:27 2023 -0600

    Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized

commit 9815d6db04c264456b7ead7fc5f735513a868c7d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:23:00 2023 -0600

    Abstract Partition Assignor

commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Jun 29 16:36:53 2023 -0700

    Refactored code to reduce format conversion time

commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 31 16:21:01 2023 -0700

    test changes and code changes

commit e1a032579129a2cd3aae1e93f46570bd57d3343c
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue May 16 09:28:57 2023 -0700

    small changes

commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu May 11 20:36:59 2023 -0700

    Squashed commit of the following:

    commit c757af5f7c630d532bfee5f6dc45aec603ad8a29
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri May 12 04:39:12 2023 +0200

        KAFKA-14752: Kafka examples improvements - demo changes (#13517)

        KAFKA-14752: Kafka examples improvements - demo changes

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 54a4067f81e1434d956ef797274f7b437fe49ea1
    Author: Kamal Chandraprakash <kchandraprakash@uber.com>
    Date:   Thu May 11 21:49:21 2023 +0530

        KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060)

        Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org>

    commit bd65db82b4bad623b0bb31398979e466978148da
    Author: Josep Prat <josep.prat@aiven.io>
    Date:   Thu May 11 17:55:26 2023 +0200

        MINOR: clean up unused methods in core utils (#13706)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org>

    commit 38adb3956979d7025d148612975cc0b82200b2e1
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Thu May 11 07:51:41 2023 -0400

        MINOR: add test tag for testDeadToDeadIllegalTransition (#13694)

        Reviewers: David Jacot <djacot@confluent.io>

    commit ee4132863553fb4fd2df715b2fbd77f349f978b8
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Thu May 11 11:19:32 2023 +0200

        KAFKA-14752: Kafka examples improvements - processor changes (#13516)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a263627adb75f1ca5c87f1482cc70b994ba49d63
    Author: Mickael Maison <mimaison@users.noreply.github.com>
    Date:   Thu May 11 11:02:45 2023 +0200

        MINOR: Remove unused methods in CoreUtils (#13170)

        Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com>

    commit 920a3601ffa7266edf12f3559cfe97c8a5929d03
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Thu May 11 04:13:29 2023 +0200

        MINOR: fix a small typo in SharedServer.scala (#13693)

        Diabled -> Disabled

    commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc
    Author: A. Sophie Blee-Goldman <sophie@confluent.io>
    Date:   Wed May 10 13:39:15 2023 -0700

        HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695)

        The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen()

        Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io>

    commit f17fb75b2de32512f14cb94a7d1bfb0f37485780
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Wed May 10 16:41:52 2023 +0200

        KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690)

        Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 4653507926a42dccda5c086fcae6278afcfc53ca
    Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
    Date:   Wed May 10 05:09:12 2023 -0700

        KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443)

        This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers.

        Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io>

    commit 625ef176ee5f167786003c2d88498632b0b7014b
    Author: Luke Chen <showuon@gmail.com>
    Date:   Wed May 10 16:40:20 2023 +0800

        MINOR: remove kraft readme link (#13691)

        The config/kraft/README.md is already removed. We should also remove the link.

        Reviewers: dengziming <dengziming1993@gmail.com>

    commit 228434d23583189cdcaa7f4a90ebb178ccc17c73
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue May 9 10:49:27 2023 -0400

        KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644)

        This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java.

        Reviewers: David Jacot <djacot@confluent.io>

    commit 59ba9dbbc927ddc8660d0d98d9422909fd306758
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 9 17:58:45 2023 +0530

        KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688)

        `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods.

        This refactoring changes no functionality other than restoring the older methods.

        Reviewers: Randall Hauch <rhauch@gmail.com>

    commit b40a7fc037bb1543c3355fad9c71570f770f5177
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon May 8 14:24:11 2023 -0700

        HOTFIX: fix broken Streams upgrade system test (#13654)

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io>

    commit 7634eee2627da39937e3112ffc58bd7cfedc98f2
    Author: David Jacot <djacot@confluent.io>
    Date:   Mon May 8 20:46:07 2023 +0200

        KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638)

        This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 86daf8ce6573eb79d6e78381dbab738055f914c4
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Mon May 8 20:09:47 2023 +0530

        KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594)

        #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR.

        Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places.

        ---------

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Mon May 8 15:24:52 2023 +0100

        KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621)

        Migrates ChangeLogReader mock in TaskManagerTest to mockito.

        Reviewer: Bruno Cadonna <cadonna@apache.org>

    commit 347238948b86882a47faee4a2916d1b01333d95f
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Mon May 8 07:36:36 2023 +0100

        KAFKA-14662: Update the ACL list in the doc (#13660)

        Added the missing ACLs to the doc.

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a556def5724efb1dc96bd2d389411a8d2f802f53
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Mon May 8 08:35:14 2023 +0200

        MINOR: Print the cause of failure for test (#13672)

        Motivation

        PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build):

        org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
        	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
        	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228)
        	at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is.
        Change

            1. Make the test a bit more robust by waiting for server startup.
            2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure.

        Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

    commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 08:33:31 2023 +0200

        MINOR: Fix producer Callback comment (#13669)

        Fixes the wrong exception name: OffsetMetadataTooLargeException.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 04:15:52 2023 +0200

        KAFKA-14752: Kafka examples improvements - producer changes (#13515)

        KAFKA-14752: Kafka examples improvements - producer changes

        Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com>

    commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Sat May 6 02:56:26 2023 +0800

        MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659)

        Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro>

    commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 14:05:20 2023 +0200

        KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312)

        Motivation

        Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit.

        In this PR, we only update varInt32 encoding/decoding.
        Changes

        This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation.
        Results

        Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java

        Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com>

    commit e34f88403159cc8381da23dafdf7e3d7403114a2
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 13:55:17 2023 +0200

        KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623)

        When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io>

    commit 0822ce0ed1a106a510930bc9ac53a266f54684d7
    Author: David Arthur <mumrah@gmail.com>
    Date:   Fri May 5 04:35:26 2023 -0400

        KAFKA-14840: Support for snapshots during ZK migration (#13461)

        This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active
        controller loaded a snapshot, it would get out of sync with the ZK state.

        In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to
        compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods
        from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and
        AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass
        and without using too much memory.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com>

    commit 97c36f3f3142580325daa1a6aadb662893390561
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 12:20:33 2023 -0700

        HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540

    commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 11:18:03 2023 -0700

        MINOR: improve QuorumController logging #13540

        When creating the QuorumController, log whether ZK migration is enabled.

        When applying a feature level record which sets the metadata version, log the metadata version enum
        rather than the numeric feature level.

        Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the
        end of replay.

        When TRACE is enabled, log every record that is replayed in QuorumController. Since some records
        may contain sensitive information, create RecordRedactor to assist in logging only what is safe to
        put in the log4j file.

        Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged
        at INFO, and additions are logged at TRACE.

        Remove SnapshotReason.java, SnapshotReasonTest.java, and
        QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to
        org.apache.kafka.image.publisher.SnapshotGenerator.

        Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu May 4 09:55:45 2023 -0700

        KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607)

        Also modifies verification to only add a partition to verify if it is transactional.

        When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different.

        Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit ea81e99e5980c807414651034a8c60426a158ca4
    Author: Philip Nee <pnee@confluent.io>
    Date:   Thu May 4 09:20:01 2023 -0700

        KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149)

        Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer.

        This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin.

        Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest.

        Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dc7819d7f1fe6b0160cd95246420ab10c335410b
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Thu May 4 11:00:33 2023 +0100

        KAFKA-14594: Move LogDirsCommand to tools module (#13122)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d46c3f259cce25c43f20fba3943d5cb34ed909ea
    Author: David Mao <47232755+splett2@users.noreply.github.com>
    Date:   Wed May 3 17:09:43 2023 -0700

        MINOR: Reduce number of threads created for integration test brokers (#13655)

        The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker.

        Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io>

    commit c08120f83f7318f15dcf14d525876d18caf6afd0
    Author: Jason Gustafson <jason@confluent.io>
    Date:   Wed May 3 15:25:32 2023 -0700

        MINOR: Allow tagged fields with version subset of flexible version range (#13551)

        The generated message types are missing a range check for the case when the tagged version range is a subset of
        the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the
        number of tags serialized.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b620c03ccf48d6d92b219cba35bb1e5e248d2547
    Author: Luke Chen <showuon@gmail.com>
    Date:   Thu May 4 01:08:25 2023 +0800

        KAFKA-14946: fix NPE when merging the deltatable (#13653)

        Fix NPE while merging the deltatable. Because it's possible that hashTier is
        not null but deltatable is null (ex: removing data), we should have null check
        while merging for deltatable like other places did. Also added tests that will
        fail without this change.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 4a0b6ebf60ed7614f042443460b490971e8662a4
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 2 23:16:46 2023 +0530

        KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657)

        Reviewers: Chris Egerton <chrise@aiven.io>

    commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue May 2 18:04:50 2023 +0200

        KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637)

        This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Tue May 2 16:39:31 2023 +0100

        MINOR: Remove unnecessary code in client/connect (#13259)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Tue May 2 14:28:42 2023 +0200

        KAFKA-14752: Kafka examples improvements - consumer changes (#13514)

        KAFKA-14752: Kafka examples improvements - consumer changes

        This is extracted from the original PR for better review.
        https://github.com/apache/kafka/pull/13492

        Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

        Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 141c76a2c904705f2cd484e96767fcb217c5db25
    Author: Bruno Cadonna <cadonna@apache.org>
    Date:   Tue May 2 14:00:34 2023 +0200

        KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529)

        1. Migrates topology builder mock in TaskManagerTest to mockito.

        2. Replaces the unit test to verify if subscribed partitions are added
        to topology metadata.

        3. Modifies signatures of methods for adding subscribed partitions to
        topology metadata to use sets instead of lists. This makes the
        intent of the methods clearer and makes the tests more portable.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org>

    commit 21af1918eafa30812a955c3c0295b9e968841cd3
    Author: Luke Chen <showuon@gmail.com>
    Date:   Tue May 2 09:54:12 2023 +0800

        MINOR: Add reason to exceptions in QuorumController (#13648)

        Saw this error message in log:

        ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController)

        The null reason is not helpful with only RejectedExecutionException. Adding the reason to it.

        Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit 4773961a44d1f0d1e11d662c7e0fc955027bced2
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon May 1 21:51:09 2023 +0530

        MINOR: Fix Javadoc for configureAdminResources  in Connect's RestServer (#13635)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io>

    commit e29942347acc70aa85d47e84e2021f9c24cd7c80
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Mon May 1 09:56:04 2023 -0400

        KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628)

        Handle migrating SCRAM records in ZK when migrating from ZK to KRaft.

        This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes.

        Reviewers: David Arthur <mumrah@gmail.com>

    commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 08:46:40 2023 -0700

        MINOR: Fixing typos in the ConsumerCoordinator (#13618)

        Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io>

    commit e55fbceb6667cc1455f7fb2e96421c85741fa7df
    Author: Anton Agestam <anton.agestam@aiven.io>
    Date:   Fri Apr 28 11:54:28 2023 +0100

        MINOR: Fix incorrect description of SessionLifetimeMs (#13649)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 02:08:32 2023 -0700

        KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550)

        This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS.

        This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method.  In short, it should no long check and validate only the highest generation.  Instead, we consider 3 cases:
        1. Member will continue to hold on to its partition if there are no other owners
        2. If there are 1+ owners to the same partition. One with the highest generation will win.
        3. If two members of the same generation hold on to the same partition.  We will log an error but remove both from the assignment. (Same with the current logic)

        Here are some important notes that lead to the patch:
        - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown.
        - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error.  This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive

        To summarize the current behavior of different errors:
        `REBALANCE_IN_PROGRESS`
        - heartbeat: requestRejoin if member state is stable
        - joinGroup: rejoin immediately
        - syncGroup: rejoin immediately
        - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress.

        `UNKNOWN_MEMBER_ID`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately
        - syncGroup:  resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        `ILLEGAL_GENERATION`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing.  then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        Reviewers: David Jacot <djacot@confluent.io>

    commit 10b3e667132934084a2d275a204a1a782c2df94e
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri Apr 28 10:32:11 2023 +0200

        KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d796480fe87fd819fc0ac560ca318759180d4644
    Author: Luke Chen <showuon@gmail.com>
    Date:   Fri Apr 28 14:35:12 2023 +0800

        KAFKA-14909: check zkMigrationReady tag before migration (#13631)

        1. add ZkMigrationReady in apiVersionsResponse
        2. check all nodes if ZkMigrationReady are ready before moving to next migration state

        Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com>

    commit c708f7ba5f4f449920cec57a5b69e84e92128b54
    Author: Colin Patrick McCabe <cmccabe@apache.org>
    Date:   Thu Apr 27 19:15:26 2023 -0700

        MINOR: remove spurious call to fatalFaultHandler (#13651)

        Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805.  We should only
        invoke the fatal fault handller if we are unable to generate the activation records. If we are
        unable to write the activation records, a controller failover should be sufficient to remedy the
        situation.

        Co-authored-by: Luke Chen showuon@gmail.com

        Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com>

    commit 056657d84d84e116ffc9460872945b4d2b479ff3
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 17:04:56 2023 -0400

        MINOR add license to reviewers.py

    commit a08f31ecfefca1a51d64137a344e62a236740e62
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 14:32:19 2023 -0400

        MINOR: Adding reviewers.py (#11096)

        This script can be used to help build the "Reviewers: " string we include in commit messages.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit 70493336171363cfa95237a0fe14ef57090553e4
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Wed Apr 26 16:10:46 2023 -0700

        KAFKA-14943: Fix ClientQuotaControlManager validation

        Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be
        used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of
        internal ZK representation details that treated them both as "client configs.")

        Add unit tests for ClientQuotaControlManager.isValidIpEntity and
        ClientQuotaControlManager.configKeysForEntityType.

        This change doesn't affect metadata record application, only input validation. If there are bad
        client quotas that are set currently, this change will not alter the current behavior (of throwing
        an exception and ignoring the bad quota).

    commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051
    Author: David Jacot <djacot@confluent.io>
    Date:   Thu Apr 27 14:05:41 2023 +0200

        KAFKA-14462; [9/N] Add RecordHelpers (#13544)

        This patch adds RecordHelpers.

        Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9
    Author: LinShunKang <linshunkang.chn@gmail.com>
    Date:   Thu Apr 27 10:44:08 2023 +0800

        KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545)

        This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
        Direct use ByteBuffer instead of byte[] to deserialize.

        Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro>

    commit c1b5c75d9271638776392822a094e9e7ef37f490
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 26 10:20:30 2023 -0400

        KAFKA-14805 KRaft controller supports pre-migration mode (#13407)

        This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode,
        the controller will only allow certain write operations. The purpose of this is to disallow metadata
        changes when the controller is waiting for the ZK migration records to be committed.

        The following ControllerWriteEvent operations are permitted in pre-migration mode

        * completeActivation
        * maybeFenceReplicas
        * writeNoOpRecord
        * processBrokerHeartbeat
        * registerBroker (only for migrating ZK brokers)
        * unregisterBroker

        Raft events and other controller events do not follow the same code path as ControllerWriteEvent,
        so they are not affected by this new behavior.

        This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState

        In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean
        MIGRATION rather than PRE_MIGRATION.

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 007c0d375a6e70aefb65f58f9f096016c4cbea16
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 26 17:53:56 2023 +0530

        KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634)

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit baf127a6633161cb52747467880b006d2f54d3bd
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Wed Apr 26 00:30:13 2023 -0700

        KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429)

        Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io>

    commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Tue Apr 25 22:39:23 2023 -0400

        KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit a7d0b3f753708a93aea92e614833f6f6e7443234
    Author: Said Boudjelda <bmscomp@gmail.com>
    Date:   Tue Apr 25 23:31:04 2023 +0200

        MINOR: Upgrade gradle to 8.1.1 (#13625)

        Also upgrade gradle plugins:
         - `org.owasp.dependencycheck` gradle plugin to version `8.2.1`
         - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1`

        Gradle release notes:
        * https://docs.gradle.org/8.1.1/release-notes.html

        Reviewers: Ismael Juma <ismael@juma.me.uk>

    commit 9a36da12b7359b7158332c541655716312efb5b3
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue Apr 25 18:50:51 2023 +0200

        KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538)

        This patch adds ConsumerGroupMember.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 4780dc773f2cd5a20fa5be38d20137745690a888
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue Apr 25 21:20:35 2023 +0530

        KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Tue Apr 25 13:28:37 2023 +0100

        KAFKA-14592: Move FeatureCommand to tools (#13459)

        KAFKA-14592: Move FeatureCommand to tools

        Reviewers: Luke Chen <showuon@gmail.com>

    commit d83a734c41c43a03aa571e602cf8535f0893fd79
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 25 12:02:24 2023 +0200

        MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626)

        The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation
        in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call.

        Reviewers:   maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com>

    commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 15:29:57 2023 -0700

        KAFKA-12446: update change encoding to use varint (#13533)

        KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>,  Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io>

    commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Mon Apr 24 17:06:26 2023 -0400

        KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit 6dcdb017327587a6943fa868595fa3488c7f7ef7
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 12:40:25 2023 -0700

        KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592)

        Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks.

        This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io>

    commit 2271e748a11919d07698ebce759dca2e3075596a
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Mon Apr 24 17:21:19 2023 +0800

        MINOR: fix zookeeper_migration_test.py (#13620)

        Reviewers: Mickael Maison <mimaison@users.noreply.github.com>

    commit c3241236296f9aaee103e475f242e32f04d4c256
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon Apr 24 14:06:20 2023 +0530

        KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 7061475445cb6314e7cf4f9848384224b14f4395
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Fri Apr 21 12:55:41 2023 -0700

        KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575)

        Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest`

        Reviewers: Josep Prat <jlprat@apache.org>

    commit ecdef88f744410039b88e90a5979078d5735aa06
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Fri Apr 21 12:48:05 2023 -0700

        MINOR: updated KS release notes for 3.5 (#13577)

        Reviewers: Walker Carlson <wcarlson@confluent.io>

    commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Fri Apr 21 15:02:06 2023 +0200

        MINOR: fix noticed typo in raft and metadata projects (#13612)

        Reviewers: Josep Prat <jlprat@apache.org>

    commit c39bf714bbaf2a632be4c2a7e446553fe40ba129
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:22:16 2023 +0200

        KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537)

        This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 2d0b816150c79057c813387bd126523b6326a1fc
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:19:04 2023 +0200

        MINOR: Move `ControllerPurgatory` to `server-common` (#13555)

        This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit df137752542c005c6998c37c03222ffbeca0f349
    Author: Purshotam Chauhan <pchauhan@confluent.io>
    Date:   Fri Apr 21 14:08:23 2023 +0530

        KAFKA-14828: Remove R/W locks using persistent data structures (#13437)

        Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only.

        In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation.

        Reviewers:  Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>,  Divij Vaidya <diviv@amazon.com>

    commit 2ee770ac7e576c85a911ccb307339be3d58a8942
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu Apr 20 10:21:26 2023 -0700

        Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)"

        This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc.

    commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu Apr 20 09:30:11 2023 -0700

        KAFKA-14904: Pending state blocked verification of transactions (#13579)

        KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction.

        There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes.

        When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions.

        The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely.

        This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 7f175feacaba4187bbb3631dff3a1330060f191e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Thu Apr 20 13:50:29 2023 +0530

        KAFKA-14586: Adding redirection for StreamsResetter (#13614)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

    commit e14dd8024adae746c50c8b7d9cd268e859669576
    Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com>
    Date:   Thu Apr 20 05:29:27 2023 +0200

        KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432)

        We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic:
        1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry.
        2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of  the client.
        3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`.

        We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them.

        Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io>

    commit f5de4daa71f0ad31aa64443c5ff43c59712feea5
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Wed Apr 19 19:45:26 2023 -0400

        KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires

        FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change
        events. However, it does not distinguish between issues related to feature changes actually failing and other
        exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration
        could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between
        these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues.
        We shutdown the broker only for FeatureCacheUpdateException.

        Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 11c8bf4826197533807b2132cfc6599ba70de1c1
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Wed Apr 19 19:34:36 2023 -0400

        KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609)

        This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter.

        Part of KIP-914.

        Reviewer: Matthias J. Sax <matthias@confluent.io>

    commit 809966a9a06664e0b521c2298fa0de834e443607
    Author: Matthew de Detrich <matthew.dedetrich@aiven.io>
    Date:   Wed Apr 19 20:54:07 2023 +0200

        KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478)

        Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address.

        Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org>

    commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 19 14:19:13 2023 -0400

        KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606)

        This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b10716e72370c4e128bddb17bcc107ccab221e47
    Author: hudeqi <1217150961@qq.com>
    Date:   Thu Apr 20 00:49:08 2023 +0800

        KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com>

    commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc
    Author: Keith Wall <kwall@redhat.com>
    Date:   Wed Apr 19 03:58:29 2023 +0100

        KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)

        Changes SocketServer to set the setReuseAddress(true) socket option.

        This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test.

        Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com>

    commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 19 00:46:03 2023 +0530

        MINOR: Fixing gradle build during compileScala and compileTestScala (#13588)

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

    commit 3388adf1b52f545e2d69edd30cdd53241b2887f3
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Tue Apr 18 11:32:27 2023 -0700

        MINOR: rename internal FK-join processor classes (#13589)

        Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>

    commit abca86511ecc9e081d676976ff6d3b845308f444
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Tue Apr 18 12:41:38 2023 -0400

        KAFKA-14881: Rework UserScramCredentialRecord (#13513)

        Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This
        is necessary to support migration from ZK, since those are the fields we stored in ZK.  Update
        latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version.  Moved
        ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more
        appropriate.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 61530d68ce83467de6190a52da37b3c0af84f0ef
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 09:37:04 2023 -0400

        KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526)

        This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields.

        Reviewers: David Jacot <djacot@confluent.io>

    commit b36a170aa3b2738177d7229859db765e0115385b
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 18 13:36:56 2023 +0200

        MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 5e9d4de748dff7b91043a9b799716ab4becdae7e
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 04:41:54 2023 -0400

        KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511)

        This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io>

    commit 454b72161a76b1687a1263157d7cc30a1bdb2506
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Tue Apr 18 09:40:14 2023 +0200

        KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557)

        KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow.
        To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread.

    commit e27926f92b1f6b34ed6731f33c712a5d0d594275
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Mon Apr 17 17:52:28 2023 -0400

        KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280)

        topic counts.

        Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change.  Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events.  We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods.

        Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>

    commit 7159f6c1a81ead277030f55f293943270346ad4e
    Author: Alyssa Huang <ahuang@confluent.io>
    Date:   Mon Apr 17 11:03:45 2023 -0700

        MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields

        Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it
        should have been. Add a regression test.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit cc88822986e4004e772f376eae47dbe08e29d137
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 10 16:29:27 2023 -0700

    small changes

commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 25 14:58:20 2023 -0700

    grammar and code comments changes

commit 5550e4934ade28f069e4ae9ab97e76982ff3401f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 17:39:48 2023 -0700

    fixed formatting stuff

commit 28ab29819bf327309ce07196a3fb0492f7fb2281
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 11:38:57 2023 -0700

    consumers -> members

commit e56ded267456bbe1c22ded0d037953545b921d1d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 11:26:23 2023 -0700

    Interface changes incorporated

commit 2d50c278c18baa11b0235330149dd8f9def57c3d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 13 14:47:33 2023 -0700

    Renamed topicIdToPartition class

commit d33bbfac053ccac77204ac4253698ed356d7c621
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 17:31:26 2023 -0700

    Added stickiness test

commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 13:28:24 2023 -0700

    Separated uniform and general builder in different files, added more tests

commit ef814867aba68fc4813359c5cef6ae155ed49a60
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:24:27 2023 -0700

    import changes

commit dffad677c6e4a33625401d621f046af6f21feb99
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:16:33 2023 -0700

    Fixed checkstyle and added test file

commit 7fd779418df3504a490c0d10f3848360a4417077
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 5 15:13:38 2023 -0700

    First draft optimised uniform assignor code

commit 309cc6141f7962e794c90d4107f460fafd51697d
Merge: 3d9264547c 2c1cf03a89
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Mon Apr 17 11:03:07 2023 -0700

    Merge branch 'trunk' into rreddy-22/KAFKA-14514

commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 10:58:16 2023 -0700

    Interface changes incorporated, added stickiness test and non-existent topic test.

commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436
Merge: a1bdb58e08 57b94ca208
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 09:49:23 2023 -0700

    Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514

    # Conflicts:
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java

commit a1bdb58e082d98b73f50860db5b35684f4eba51b
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 20:01:59 2023 -0700

    Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc.

commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 11:28:40 2023 -0700

    Addressed some PR comments

commit 918d262ae52892c13766649db847ede6928921f8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7369db3744b4c89f625f6b73db331eaf0512c4eb
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit a925b02dd3d78c8b9c1415c083f0544edcb404fe
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit 48a982232c6f1ff371cb44333fd12d6b37ef381a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation

commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 13:00:07 2023 -0700

    Removed topicIdToPartition class, changed attribute names and added java doc for getter methods

commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 15:09:44 2023 -0700

    Made all attributes private and added getter methods, changed names according to PR comments

commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:25:33 2023 -0700

    Added toString method and hash

commit 5065109332845541c5591a52683f21d37982d80a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 13:43:26 2023 -0700

    Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure

commit 57b94ca208cb374b22d61eb044ef8004ea09479f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7352cbb5683f676382f56e6ae5c926f3cae98824
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0
Merge: 4b321aa537 637bc92ba1
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Thu Apr 6 10:37:58 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514

commit 4b321aa5374427a15346be9b87b7c545ccb7db61
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 74105291cc687fe41b75de692fc624ab1335341a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit 3df34605e078beb9babcbeadc7b119a70da63796
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation
rreddy-22 added a commit to rreddy-22/kafka-rreddy that referenced this pull request Jul 25, 2023
@rreddy-22
Squashed commit of the following:
1382b1e 
commit 71f8488e5bf30af6f4a465d1fac52ccb9a341396
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 25 15:22:10 2023 -0700

    Deleted PartitionMetadata and added TopicMetadata to the SubscribedTopicMetadata, PR comments

commit 5a935324c11e6f6309ee5451f90ed5052276a734
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 25 10:51:11 2023 -0700

    Minor

commit 1c955b5e251aedc174cf28f43037f08162d89b49
Merge: 2d40125fa8 81f1ccd7a2
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 13:02:54 2023 -0700

    Merge branch 'rreddy-22/Rack-Assignor-Interface-Changes' of github.com:rreddy-22/kafka-rreddy into rreddy-22/Rack-Assignor-Interface-Changes

commit 81f1ccd7a25d85eaf1cff713df402e0100d4b4a3
Merge: a39e76f63f 84691b11f6
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Mon Jul 24 12:56:40 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes

commit 2d40125fa8ba16b4f07e6bfe410ceacfd8a112e8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 11:45:21 2023 -0700

    minor

commit a39e76f63f312ecacdcebd2c5015777e160ca0b3
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 10:04:10 2023 -0700

    reverted reviewers.py changes

commit c0b464c8936f1df4f6540b8abe7c1fa88fce6e81
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Jul 24 00:02:03 2023 -0700

    reverted grammar changes

commit 6fe72dfce921ab9d77ba6b28d4104c5d92c6dbf3
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 23:56:35 2023 -0700

    minor

commit a30a53463e55b72699617360d1c888b7ee310336
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 23:55:35 2023 -0700

    minor

commit 57b9a6233a4f697f285b85e6394d3dd407f16404
Merge: 061dac797b 4981fa939d
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Sun Jul 23 19:52:07 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/Rack-Assignor-Interface-Changes

commit 061dac797bae56d91659fbc8d7900117eb10a866
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:43:12 2023 -0700

    Moved SubscribedTopicMetadata to the consumer package

commit 6650ea5cd707f03713fc2c69ba76fafd56d886b8
Merge: 22f2df02de faafed25a1
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:33:17 2023 -0700

    Merge remote-tracking branch 'origin/rreddy-22/Rack-Assignor-Interface-Changes' into rreddy-22/Rack-Assignor-Interface-Changes

commit 22f2df02deabe1a6b5b8b8cceb432486b9ef4470
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Sun Jul 23 19:25:11 2023 -0700

    minor changes

commit faafed25a10d7ff79fb8a24ab62aa222c0e6516a
Merge: 4511683d4e 1bf73d89d0
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Fri Jul 21 12:50:59 2023 -0700

    Merge branch 'trunk' into rreddy-22/Rack-Assignor-Interface-Changes

commit 4511683d4ef489cb979769ff6775dfe0250cdeeb
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 21 03:09:27 2023 -0700

    Changes based on PR comments

commit 579bb948fcc78afd163410a5bc43832bc87aaf71
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Jul 19 14:45:18 2023 -0700

    New interfaces and classes added to facilitate passing rack information to the assignor, modified tests to incorporate changes

commit f68471a99f2e0260b36047a7596b0f9de48794ad
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Jul 13 11:55:20 2023 -0700

    Small edits

commit c50f3bdbf79f1ca127355d2a9002b1fdbf0baa2f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 11 16:30:32 2023 -0700

    Removed extra lines

commit 478488e0943aa879b27eec5ae5eb79a23dd9cab3
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Jul 11 16:21:54 2023 -0700

    Added rack interface changes, added a new TopicAndClusterMetadata class to handle topic and cluster images and added an AbstractPartitionAssignor

commit ce540f943b8dae0786a87e51cafacb66efd1e64d
Merge: 9815d6db04 fd5b300b57
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:45:27 2023 -0600

    Merge remote-tracking branch 'upstream/trunk' into rreddy-22/KAFKA-14515-Optimized

commit 9815d6db04c264456b7ead7fc5f735513a868c7d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Jul 7 12:23:00 2023 -0600

    Abstract Partition Assignor

commit e9ebc4f0a48c5ea08391377ffcdb01c375f2be14
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Jun 29 16:36:53 2023 -0700

    Refactored code to reduce format conversion time

commit 931bcf7f82f9ef9a1b2f0be4b7d92c7f226ace08
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 31 16:21:01 2023 -0700

    test changes and code changes

commit e1a032579129a2cd3aae1e93f46570bd57d3343c
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue May 16 09:28:57 2023 -0700

    small changes

commit 1c2475b98f08b92bf876f2ca4eb979a2fca5848a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu May 11 20:36:59 2023 -0700

    Squashed commit of the following:

    commit c757af5f7c630d532bfee5f6dc45aec603ad8a29
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri May 12 04:39:12 2023 +0200

        KAFKA-14752: Kafka examples improvements - demo changes (#13517)

        KAFKA-14752: Kafka examples improvements - demo changes

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 54a4067f81e1434d956ef797274f7b437fe49ea1
    Author: Kamal Chandraprakash <kchandraprakash@uber.com>
    Date:   Thu May 11 21:49:21 2023 +0530

        KAFKA-14559: Fix JMX tool to handle the object names with wildcard and optional attributes (#13060)

        Reviewers: Federico Valeri <fedevaleri@gmail.com>, Satish Duggana <satishd@apache.org>

    commit bd65db82b4bad623b0bb31398979e466978148da
    Author: Josep Prat <josep.prat@aiven.io>
    Date:   Thu May 11 17:55:26 2023 +0200

        MINOR: clean up unused methods in core utils (#13706)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Mickael Maison <mimaison@apache.org>

    commit 38adb3956979d7025d148612975cc0b82200b2e1
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Thu May 11 07:51:41 2023 -0400

        MINOR: add test tag for testDeadToDeadIllegalTransition (#13694)

        Reviewers: David Jacot <djacot@confluent.io>

    commit ee4132863553fb4fd2df715b2fbd77f349f978b8
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Thu May 11 11:19:32 2023 +0200

        KAFKA-14752: Kafka examples improvements - processor changes (#13516)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a263627adb75f1ca5c87f1482cc70b994ba49d63
    Author: Mickael Maison <mimaison@users.noreply.github.com>
    Date:   Thu May 11 11:02:45 2023 +0200

        MINOR: Remove unused methods in CoreUtils (#13170)

        Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com>

    commit 920a3601ffa7266edf12f3559cfe97c8a5929d03
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Thu May 11 04:13:29 2023 +0200

        MINOR: fix a small typo in SharedServer.scala (#13693)

        Diabled -> Disabled

    commit 6d2ad4a38340176184e3f19027b5e0e024c1f2cc
    Author: A. Sophie Blee-Goldman <sophie@confluent.io>
    Date:   Wed May 10 13:39:15 2023 -0700

        HOTFIX: fix the VersionedKeyValueToBytesStoreAdapter#isOpen API (#13695)

        The VersionedKeyValueToBytesStoreAdapter#isOpen API accidentally returns the value of inner.persistent() when it should be returning inner.isOpen()

        Reviewers: Matthias J. Sax <mjsax@apache.org>, Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>, Victoria Xia <victoria.xia@confluent.io>

    commit f17fb75b2de32512f14cb94a7d1bfb0f37485780
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Wed May 10 16:41:52 2023 +0200

        KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics (#13690)

        Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 4653507926a42dccda5c086fcae6278afcfc53ca
    Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
    Date:   Wed May 10 05:09:12 2023 -0700

        KAFKA-14514; Add Range Assignor on the Server (KIP-848) (#13443)

        This patch adds the RangeAssignor on the server for KIP-848. This range assignor is very different from the old client side implementation. We added functionality to make these assignments sticky while also inheriting crucial properties of the range assignor such as facilitating joins and distributing partitions of a topic somewhat equally amongst its subscribers.

        Reviewers: Philip Nee <philipnee@gmail.com>, Jeff Kim <jeff.kim@confluent.io>, David Jacot <djacot@confluent.io>

    commit 625ef176ee5f167786003c2d88498632b0b7014b
    Author: Luke Chen <showuon@gmail.com>
    Date:   Wed May 10 16:40:20 2023 +0800

        MINOR: remove kraft readme link (#13691)

        The config/kraft/README.md is already removed. We should also remove the link.

        Reviewers: dengziming <dengziming1993@gmail.com>

    commit 228434d23583189cdcaa7f4a90ebb178ccc17c73
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue May 9 10:49:27 2023 -0400

        KAFKA-14500; [1/N] Rewrite MemberMetadata in Java (#13644)

        This patch adds GenericGroupMember which is a rewrite of MemberMetadata in Java.

        Reviewers: David Jacot <djacot@confluent.io>

    commit 59ba9dbbc927ddc8660d0d98d9422909fd306758
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 9 17:58:45 2023 +0530

        KAFKA-14974: Restore backward compatibility in KafkaBasedLog (#13688)

        `KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods.

        This refactoring changes no functionality other than restoring the older methods.

        Reviewers: Randall Hauch <rhauch@gmail.com>

    commit b40a7fc037bb1543c3355fad9c71570f770f5177
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon May 8 14:24:11 2023 -0700

        HOTFIX: fix broken Streams upgrade system test (#13654)

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, John Roesler <john@confluent.io>

    commit 7634eee2627da39937e3112ffc58bd7cfedc98f2
    Author: David Jacot <djacot@confluent.io>
    Date:   Mon May 8 20:46:07 2023 +0200

        KAFKA-14462; [11/N] Add CurrentAssignmentBuilder (#13638)

        This patch adds the `CurrentAssignmentBuilder` class which encapsulates the reconciliation engine of the consumer group protocol. Given the current state of a member and a desired or target assignment state, the state machine takes the necessary steps to converge the member to its desired state.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 86daf8ce6573eb79d6e78381dbab738055f914c4
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Mon May 8 20:09:47 2023 +0530

        KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime (#13594)

        #13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR.

        Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places.

        ---------

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit 2b98f8553ba12ab5d8cb88f5cd0d1198cb424df6
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Mon May 8 15:24:52 2023 +0100

        KAFKA-14133: Migrate ChangeLogReader mock in TaskManagerTest to Mockito (#13621)

        Migrates ChangeLogReader mock in TaskManagerTest to mockito.

        Reviewer: Bruno Cadonna <cadonna@apache.org>

    commit 347238948b86882a47faee4a2916d1b01333d95f
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Mon May 8 07:36:36 2023 +0100

        KAFKA-14662: Update the ACL list in the doc (#13660)

        Added the missing ACLs to the doc.

        Reviewers: Luke Chen <showuon@gmail.com>

    commit a556def5724efb1dc96bd2d389411a8d2f802f53
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Mon May 8 08:35:14 2023 +0200

        MINOR: Print the cause of failure for test (#13672)

        Motivation

        PlaintextAdminIntegrationTest fails in a flaky manner with the follow trace (e.g. in this build):

        org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
        	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
        	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
        	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:228)
        	at kafka.api.PlaintextAdminIntegrationTest.testElectUncleanLeadersForOnePartition(PlaintextAdminIntegrationTest.scala:1583)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        The std output doesn't contain useful information that we could use to debug the cause of failure. This is because the test, currently, validates if there is an exception and fails when one is present. It does not print what the exception is.
        Change

            1. Make the test a bit more robust by waiting for server startup.
            2. Fail the test with the actual unexpected exception which will help in debugging the cause of failure.

        Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>

    commit 2607e0edb7c165bf2c340e81c2a39d7bb3b63fbf
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 08:33:31 2023 +0200

        MINOR: Fix producer Callback comment (#13669)

        Fixes the wrong exception name: OffsetMetadataTooLargeException.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 78090bb4cdd2494f0b720d34e17ee0cc645fc399
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Mon May 8 04:15:52 2023 +0200

        KAFKA-14752: Kafka examples improvements - producer changes (#13515)

        KAFKA-14752: Kafka examples improvements - producer changes

        Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <christololov@gmail.com>

    commit 6e7144ac24973afdb71ef59a63c6bacbbb1d2714
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Sat May 6 02:56:26 2023 +0800

        MINOR: add docs to remind reader that impl of ConsumerPartitionAssign… (#13659)

        Reviewers: David Jacot <djacot@confluent.io>, Kirk True <kirk@kirktrue.pro>

    commit 6bcc497c36a1aef19204b1bfe3b17a8c1c84c059
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 14:05:20 2023 +0200

        KAFKA-14766: Improve performance of VarInt encoding and decoding (#13312)

        Motivation

        Reading/writing the protocol buffer varInt32 and varInt64 (also called varLong in our code base) is in the hot path of data plane code in Apache Kafka. We read multiple varInt in a record and in long. Hence, even a minor change in performance could extrapolate to larger performance benefit.

        In this PR, we only update varInt32 encoding/decoding.
        Changes

        This change uses loop unrolling and reduces the amount of repetition of calculations. Based on the empirical results from the benchmark, the code has been modified to pick up the best implementation.
        Results

        Performance has been evaluated using JMH benchmarks on JDK 17.0.6. Various implementations have been added in the benchmark and benchmarking has been done for different sizes of varints and varlongs. The benchmark for various implementations have been added at ByteUtilsBenchmark.java

        Reviewers: Ismael Juma <mlists@juma.me.uk>, Luke Chen <showuon@gmail.com>, Alexandre Dupriez <alexandre.dupriez@gmail.com>

    commit e34f88403159cc8381da23dafdf7e3d7403114a2
    Author: Divij Vaidya <diviv@amazon.com>
    Date:   Fri May 5 13:55:17 2023 +0200

        KAFKA-14926: Remove metrics on Log Cleaner shutdown (#13623)

        When Log cleaning is shutdown, it doesn't remove metrics that were registered to `KafkaYammerMetrics.defaultRegistry()` which has one instance per server. Log cleaner's lifecycle is associated with lifecycle of `LogManager` and hence, there is no possibility where log cleaner will be shutdown but the broker won't. Broker shutdown will close the `jmxReporter` and hence, there is no current metric leak here. The motivation for this code change is to "do the right thing" from a code hygiene perspective.

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Kirk True <kirk@mustardgrain.com>, David Jacot <djacot@confluent.io>

    commit 0822ce0ed1a106a510930bc9ac53a266f54684d7
    Author: David Arthur <mumrah@gmail.com>
    Date:   Fri May 5 04:35:26 2023 -0400

        KAFKA-14840: Support for snapshots during ZK migration (#13461)

        This patch adds support for handling metadata snapshots while in dual-write mode. Prior to this change, if the active
        controller loaded a snapshot, it would get out of sync with the ZK state.

        In order to reconcile the snapshot state with ZK, several methods were added to scan through the metadata in ZK to
        compute differences with the MetadataImage. Since this introduced a lot of code, I opted to split out a lot of methods
        from ZkMigrationClient into their own client interfaces, such as TopicMigrationClient, ConfigMigrationClient, and
        AclMigrationClient. Each of these has some iterator method that lets the caller examine the ZK state in a single pass
        and without using too much memory.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>, Luke Chen <showuon@gmail.com>

    commit 97c36f3f3142580325daa1a6aadb662893390561
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 12:20:33 2023 -0700

        HOTFIX: fix file deletions left out of MINOR: improve QuorumController logging #13540

    commit 63f9f23ec0aaa62f0da93ebc42934f5fce743ddb
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu May 4 11:18:03 2023 -0700

        MINOR: improve QuorumController logging #13540

        When creating the QuorumController, log whether ZK migration is enabled.

        When applying a feature level record which sets the metadata version, log the metadata version enum
        rather than the numeric feature level.

        Improve the logging when we replay snapshots in QuorumController. Log both the beginning and the
        end of replay.

        When TRACE is enabled, log every record that is replayed in QuorumController. Since some records
        may contain sensitive information, create RecordRedactor to assist in logging only what is safe to
        put in the log4j file.

        Add logging to ControllerPurgatory. Successful completions are logged at DEBUG; failures are logged
        at INFO, and additions are logged at TRACE.

        Remove SnapshotReason.java, SnapshotReasonTest.java, and
        QuorumController#generateSnapshotScheduled. They are deadcode now that snapshot generation moved to
        org.apache.kafka.image.publisher.SnapshotGenerator.

        Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit ffd814d25fb97f2ee0b73000788c93ec1d5b9bff
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu May 4 09:55:45 2023 -0700

        KAFKA-14916: Fix code that assumes transactional ID implies all records are transactional (#13607)

        Also modifies verification to only add a partition to verify if it is transactional.

        When verifying we look at all the transactional producer IDs and throw INVALID_RECORD on the request if one is different.

        Reviewers: Kirk True <ktrue@confluent.io>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit ea81e99e5980c807414651034a8c60426a158ca4
    Author: Philip Nee <pnee@confluent.io>
    Date:   Thu May 4 09:20:01 2023 -0700

        KAFKA-13668: Retry upon missing initProducerId due to authorization error (#12149)

        Producers used to throw a fatal error upon failing initProducerId, which can be caused by authorization errors. In this case, the user will need to instantiate a producer.

        This PR makes authorization errors non-fatal so that the user can retry until the permission is fixed by an admin.

        Here we first transition the producer to the ABORTABLE state, then to the UNINITIALIZED state (so that the producer is recoverable). Upon the subsequent send, the producer will transition to INITIALIZING and attempt to send another InitProducerIdRequest.

        Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dc7819d7f1fe6b0160cd95246420ab10c335410b
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Thu May 4 11:00:33 2023 +0100

        KAFKA-14594: Move LogDirsCommand to tools module (#13122)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d46c3f259cce25c43f20fba3943d5cb34ed909ea
    Author: David Mao <47232755+splett2@users.noreply.github.com>
    Date:   Wed May 3 17:09:43 2023 -0700

        MINOR: Reduce number of threads created for integration test brokers (#13655)

        The integration tests seem to create an unnecessarily large number of threads. This reduces the number of threads created per integration test harness broker.

        Reviewers: Luke Chen <showuon@gmail.com>. Justine Olshan <jolshan@confluent.io>

    commit c08120f83f7318f15dcf14d525876d18caf6afd0
    Author: Jason Gustafson <jason@confluent.io>
    Date:   Wed May 3 15:25:32 2023 -0700

        MINOR: Allow tagged fields with version subset of flexible version range (#13551)

        The generated message types are missing a range check for the case when the tagged version range is a subset of
        the flexible version range. This causes the tagged field count, which is computed correctly, to conflict with the
        number of tags serialized.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b620c03ccf48d6d92b219cba35bb1e5e248d2547
    Author: Luke Chen <showuon@gmail.com>
    Date:   Thu May 4 01:08:25 2023 +0800

        KAFKA-14946: fix NPE when merging the deltatable (#13653)

        Fix NPE while merging the deltatable. Because it's possible that hashTier is
        not null but deltatable is null (ex: removing data), we should have null check
        while merging for deltatable like other places did. Also added tests that will
        fail without this change.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 4a0b6ebf60ed7614f042443460b490971e8662a4
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue May 2 23:16:46 2023 +0530

        KAFKA-14876: Document the new 'PUT /connectors/{name}/stop' REST API for Connect (#13657)

        Reviewers: Chris Egerton <chrise@aiven.io>

    commit 16fc8e1cfff6f0ac29209704a079b0ddcbd0625e
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue May 2 18:04:50 2023 +0200

        KAFKA-14462; [10/N] Add TargetAssignmentBuilder (#13637)

        This patch adds TargetAssignmentBuilder. It is responsible for computing a target assignment for a given group.

        Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit f44ee4fab7ef7adf715ecf2b96defa5cc8311949
    Author: Christo Lolov <lolovc@amazon.com>
    Date:   Tue May 2 16:39:31 2023 +0100

        MINOR: Remove unnecessary code in client/connect (#13259)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 33012b5ec34305a5133eb6e9e2fb6e8c3178f3b3
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Tue May 2 14:28:42 2023 +0200

        KAFKA-14752: Kafka examples improvements - consumer changes (#13514)

        KAFKA-14752: Kafka examples improvements - consumer changes

        This is extracted from the original PR for better review.
        https://github.com/apache/kafka/pull/13492

        Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

        Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>

    commit 141c76a2c904705f2cd484e96767fcb217c5db25
    Author: Bruno Cadonna <cadonna@apache.org>
    Date:   Tue May 2 14:00:34 2023 +0200

        KAFKA-14133: Migrate topology builder mock in TaskManagerTest to mockito (#13529)

        1. Migrates topology builder mock in TaskManagerTest to mockito.

        2. Replaces the unit test to verify if subscribed partitions are added
        to topology metadata.

        3. Modifies signatures of methods for adding subscribed partitions to
        topology metadata to use sets instead of lists. This makes the
        intent of the methods clearer and makes the tests more portable.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Matthias J. Sax <mjsax@apache.org>

    commit 21af1918eafa30812a955c3c0295b9e968841cd3
    Author: Luke Chen <showuon@gmail.com>
    Date:   Tue May 2 09:54:12 2023 +0800

        MINOR: Add reason to exceptions in QuorumController (#13648)

        Saw this error message in log:

        ERROR [QuorumController id=1] writeNoOpRecord: unable to start processing because of RejectedExecutionException. Reason: null (org.apache.kafka.controller.QuorumController)

        The null reason is not helpful with only RejectedExecutionException. Adding the reason to it.

        Reviewers: David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit 4773961a44d1f0d1e11d662c7e0fc955027bced2
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon May 1 21:51:09 2023 +0530

        MINOR: Fix Javadoc for configureAdminResources  in Connect's RestServer (#13635)

        Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io>

    commit e29942347acc70aa85d47e84e2021f9c24cd7c80
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Mon May 1 09:56:04 2023 -0400

        KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628)

        Handle migrating SCRAM records in ZK when migrating from ZK to KRaft.

        This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes.

        Reviewers: David Arthur <mumrah@gmail.com>

    commit 64ebbc577de757830b2f26ee3c8c7b1ddf10f86c
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 08:46:40 2023 -0700

        MINOR: Fixing typos in the ConsumerCoordinator (#13618)

        Reviewers: Divij Vaidya <diviv@amazon.com>, Christo Lolov <lolovc@amazon.com>, David Jacot <djacot@confluent.io>

    commit e55fbceb6667cc1455f7fb2e96421c85741fa7df
    Author: Anton Agestam <anton.agestam@aiven.io>
    Date:   Fri Apr 28 11:54:28 2023 +0100

        MINOR: Fix incorrect description of SessionLifetimeMs (#13649)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit c6ad151ac3bac0d8d1d6985d230eacaa170b8984
    Author: Philip Nee <pnee@confluent.io>
    Date:   Fri Apr 28 02:08:32 2023 -0700

        KAFKA-14639: A single partition may be revoked and assign during a single round of rebalance (#13550)

        This is a really long story, but the incident started in KAFKA-13419 when we observed a member sending out a topic partition owned from the previous generation when a member missed a rebalance cycle due to REBALANCE_IN_PROGRESS.

        This patch changes the AbstractStickyAssignor.AllSubscriptionsEqual method.  In short, it should no long check and validate only the highest generation.  Instead, we consider 3 cases:
        1. Member will continue to hold on to its partition if there are no other owners
        2. If there are 1+ owners to the same partition. One with the highest generation will win.
        3. If two members of the same generation hold on to the same partition.  We will log an error but remove both from the assignment. (Same with the current logic)

        Here are some important notes that lead to the patch:
        - If a member is kicked out of the group, and `UNKNOWN_MEMBER_ID` will be thrown.
        - It seems to be a common situation that members are late to joinGroup and therefore get `REBALANCE_IN_PROGRESS` error.  This is why we don't want to reset generation because it might cause lots of revocations and can be disruptive

        To summarize the current behavior of different errors:
        `REBALANCE_IN_PROGRESS`
        - heartbeat: requestRejoin if member state is stable
        - joinGroup: rejoin immediately
        - syncGroup: rejoin immediately
        - commit: requestRejoin and fail the commit. Raise this exception if the generation is staled, i.e. another rebalance is already in progress.

        `UNKNOWN_MEMBER_ID`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - joinGroup: resetStateAndRejoin if generation unchanged, otherwise rejoin immediately
        - syncGroup:  resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        `ILLEGAL_GENERATION`
        - heartbeat: resetStateAndRejoinif generation hasn't changed. otherwise, ignore
        - syncGroup: raised the exception if generation has been resetted or the member hasn't completed rebalancing.  then resetStateAndRejoin if generation unchanged, otherwise rejoin immediately

        Reviewers: David Jacot <djacot@confluent.io>

    commit 10b3e667132934084a2d275a204a1a782c2df94e
    Author: Federico Valeri <fedevaleri@gmail.com>
    Date:   Fri Apr 28 10:32:11 2023 +0200

        KAFKA-14584: Deprecate StateChangeLogMerger tool (#13171)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit d796480fe87fd819fc0ac560ca318759180d4644
    Author: Luke Chen <showuon@gmail.com>
    Date:   Fri Apr 28 14:35:12 2023 +0800

        KAFKA-14909: check zkMigrationReady tag before migration (#13631)

        1. add ZkMigrationReady in apiVersionsResponse
        2. check all nodes if ZkMigrationReady are ready before moving to next migration state

        Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com>

    commit c708f7ba5f4f449920cec57a5b69e84e92128b54
    Author: Colin Patrick McCabe <cmccabe@apache.org>
    Date:   Thu Apr 27 19:15:26 2023 -0700

        MINOR: remove spurious call to fatalFaultHandler (#13651)

        Remove a spurious call to fatalFaultHandler accidentally introduced by KAFKA-14805.  We should only
        invoke the fatal fault handller if we are unable to generate the activation records. If we are
        unable to write the activation records, a controller failover should be sufficient to remedy the
        situation.

        Co-authored-by: Luke Chen showuon@gmail.com

        Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com>

    commit 056657d84d84e116ffc9460872945b4d2b479ff3
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 17:04:56 2023 -0400

        MINOR add license to reviewers.py

    commit a08f31ecfefca1a51d64137a344e62a236740e62
    Author: David Arthur <mumrah@gmail.com>
    Date:   Thu Apr 27 14:32:19 2023 -0400

        MINOR: Adding reviewers.py (#11096)

        This script can be used to help build the "Reviewers: " string we include in commit messages.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit 70493336171363cfa95237a0fe14ef57090553e4
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Wed Apr 26 16:10:46 2023 -0700

        KAFKA-14943: Fix ClientQuotaControlManager validation

        Don't allow setting negative or zero values for quotas. Don't allow SCRAM mechanism names to be
        used as client quota names. SCRAM mechanisms are not client quotas. (The confusion arose because of
        internal ZK representation details that treated them both as "client configs.")

        Add unit tests for ClientQuotaControlManager.isValidIpEntity and
        ClientQuotaControlManager.configKeysForEntityType.

        This change doesn't affect metadata record application, only input validation. If there are bad
        client quotas that are set currently, this change will not alter the current behavior (of throwing
        an exception and ignoring the bad quota).

    commit 8bde4e79cdeea7761b54e24516a7d2cc9f52e051
    Author: David Jacot <djacot@confluent.io>
    Date:   Thu Apr 27 14:05:41 2023 +0200

        KAFKA-14462; [9/N] Add RecordHelpers (#13544)

        This patch adds RecordHelpers.

        Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit dd6690a7a0a565a681c52dbfe0c7c89875bdf8c9
    Author: LinShunKang <linshunkang.chn@gmail.com>
    Date:   Thu Apr 27 10:44:08 2023 +0800

        KAFKA-14944: Reduce CompletedFetch#parseRecord() memory copy (#12545)

        This implements KIP-863: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035
        Direct use ByteBuffer instead of byte[] to deserialize.

        Reviewers: Luke Chen <showuon@gmail.com>, Kirk True <kirk@kirktrue.pro>

    commit c1b5c75d9271638776392822a094e9e7ef37f490
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 26 10:20:30 2023 -0400

        KAFKA-14805 KRaft controller supports pre-migration mode (#13407)

        This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode,
        the controller will only allow certain write operations. The purpose of this is to disallow metadata
        changes when the controller is waiting for the ZK migration records to be committed.

        The following ControllerWriteEvent operations are permitted in pre-migration mode

        * completeActivation
        * maybeFenceReplicas
        * writeNoOpRecord
        * processBrokerHeartbeat
        * registerBroker (only for migrating ZK brokers)
        * unregisterBroker

        Raft events and other controller events do not follow the same code path as ControllerWriteEvent,
        so they are not affected by this new behavior.

        This patch also add a new metric as defined in KIP-868: kafka.controller:type=KafkaController,name=ZkMigrationState

        In order to support upgrades from 3.4.0, this patch also redefines the enum value of value 1 to mean
        MIGRATION rather than PRE_MIGRATION.

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 007c0d375a6e70aefb65f58f9f096016c4cbea16
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 26 17:53:56 2023 +0530

        KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure (#13634)

        Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>

        Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

    commit baf127a6633161cb52747467880b006d2f54d3bd
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Wed Apr 26 00:30:13 2023 -0700

        KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication (#13429)

        Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io>

    commit ced1f62c1b1cc0d547dc31fbce538885c29ed1ef
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Tue Apr 25 22:39:23 2023 -0400

        KAFKA-14834: [13/N] Docs updates for versioned store semantics (#13622)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit a7d0b3f753708a93aea92e614833f6f6e7443234
    Author: Said Boudjelda <bmscomp@gmail.com>
    Date:   Tue Apr 25 23:31:04 2023 +0200

        MINOR: Upgrade gradle to 8.1.1 (#13625)

        Also upgrade gradle plugins:
         - `org.owasp.dependencycheck` gradle plugin to version `8.2.1`
         - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1`

        Gradle release notes:
        * https://docs.gradle.org/8.1.1/release-notes.html

        Reviewers: Ismael Juma <ismael@juma.me.uk>

    commit 9a36da12b7359b7158332c541655716312efb5b3
    Author: David Jacot <djacot@confluent.io>
    Date:   Tue Apr 25 18:50:51 2023 +0200

        KAFKA-14462; [8/N] Add ConsumerGroupMember (#13538)

        This patch adds ConsumerGroupMember.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 4780dc773f2cd5a20fa5be38d20137745690a888
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Tue Apr 25 21:20:35 2023 +0530

        KAFKA-14933: Document Connect's log level REST APIs from KIP-495 (#13636)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>

    commit ea540fa40042c5e2d808cc4dfc71c71f7466fbe4
    Author: Gantigmaa Selenge <39860586+tinaselenge@users.noreply.github.com>
    Date:   Tue Apr 25 13:28:37 2023 +0100

        KAFKA-14592: Move FeatureCommand to tools (#13459)

        KAFKA-14592: Move FeatureCommand to tools

        Reviewers: Luke Chen <showuon@gmail.com>

    commit d83a734c41c43a03aa571e602cf8535f0893fd79
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 25 12:02:24 2023 +0200

        MINOR: only set sslEngine#setUseClientMode to false once when ssl mode is server (#13626)

        The sslEngine.setUseClientMode(false) was duplicated when ssl mode is server during SSLEngine creation
        in DefaultSslEngineFactory.java. The patch attemps to remove the duplicated call.

        Reviewers:   maulin-vasavada <maulin.vasavada@gmail.com>, Divij Vaidya <diviv@amazon.com>, Manikumar Reddy <manikumar.reddy@gmail.com>

    commit 2557a4b842b07ac796193bd9a3ef6b724dc995cf
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 15:29:57 2023 -0700

        KAFKA-12446: update change encoding to use varint (#13533)

        KIP-904 had the goal in mind to save space when encoding the size on a byte array. However, using UINT32 does not achieve this goal. This PR changes the encoding to VARINT instead.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>,  Farooq Qaiser <fqaiser94@gmail.com>, Walker Carlson <wcarlson@confluent.io>

    commit ab8f2850973b1e9fd548d5b7b8eae458fdd26402
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Mon Apr 24 17:06:26 2023 -0400

        KAFKA-14834: [12/N] Minor code cleanups relating to versioned stores (#13615)

        Reviewers: Matthias J. Sax <matthias@confluent.io>

    commit 6dcdb017327587a6943fa868595fa3488c7f7ef7
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Mon Apr 24 12:40:25 2023 -0700

        KAFKA-14862: Outer stream-stream join does not output all results with multiple input partitions (#13592)

        Stream-stream outer join, uses a "shared time tracker" to track stream-time progress for left and right input in a single place. This time tracker is incorrectly shared across tasks.

        This PR introduces a supplier to create a "shared time tracker" object per task, to be shared between the left and right join processors.

        Reviewers: Victoria Xia <victoria.xia@confluent.io>, Bruno Cadonna <bruno@confluent.io>, Walker Carlson <wcarlson@confluent.io>

    commit 2271e748a11919d07698ebce759dca2e3075596a
    Author: Chia-Ping Tsai <chia7712@gmail.com>
    Date:   Mon Apr 24 17:21:19 2023 +0800

        MINOR: fix zookeeper_migration_test.py (#13620)

        Reviewers: Mickael Maison <mimaison@users.noreply.github.com>

    commit c3241236296f9aaee103e475f242e32f04d4c256
    Author: Yash Mayya <yash.mayya@gmail.com>
    Date:   Mon Apr 24 14:06:20 2023 +0530

        KAFKA-14876: Document the new 'GET /connectors/{name}/offsets' REST API for Connect (#13587)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>

    commit 7061475445cb6314e7cf4f9848384224b14f4395
    Author: Greg Harris <greg.harris@aiven.io>
    Date:   Fri Apr 21 12:55:41 2023 -0700

        KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts (#13575)

        Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest`

        Reviewers: Josep Prat <jlprat@apache.org>

    commit ecdef88f744410039b88e90a5979078d5735aa06
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Fri Apr 21 12:48:05 2023 -0700

        MINOR: updated KS release notes for 3.5 (#13577)

        Reviewers: Walker Carlson <wcarlson@confluent.io>

    commit dd63d88ac3ea7a9a55a6dacf9c5473e939322a55
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Fri Apr 21 15:02:06 2023 +0200

        MINOR: fix noticed typo in raft and metadata projects (#13612)

        Reviewers: Josep Prat <jlprat@apache.org>

    commit c39bf714bbaf2a632be4c2a7e446553fe40ba129
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:22:16 2023 +0200

        KAFKA-14462; [7/N] Add ClientAssignor, Assignment, TopicMetadata and VersionedMetadata (#13537)

        This patch adds ClientAssignor, Assignment, TopicMetadata and VersionedMetadata classes.

        Reviewers: Christo Lolov <lolovc@amazon.com>, Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io>

    commit 2d0b816150c79057c813387bd126523b6326a1fc
    Author: David Jacot <djacot@confluent.io>
    Date:   Fri Apr 21 11:19:04 2023 +0200

        MINOR: Move `ControllerPurgatory` to `server-common` (#13555)

        This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org>

    commit df137752542c005c6998c37c03222ffbeca0f349
    Author: Purshotam Chauhan <pchauhan@confluent.io>
    Date:   Fri Apr 21 14:08:23 2023 +0530

        KAFKA-14828: Remove R/W locks using persistent data structures (#13437)

        Currently, StandardAuthorizer uses a R/W lock for maintaining the consistency of data. For the clusters with very high traffic, we will typically see an increase in latencies whenever a write operation comes. The intent of this PR is to get rid of the R/W lock with the help of immutable or persistent collections. Basically, new object references are used to hold the intermediate state of the write operation. After the completion of the operation, the main reference to the cache is changed to point to the new object. Also, for the read operation, the code is changed such that all accesses to the cache for a single read operation are done to a particular cache object only.

        In the PR description, you can find the performance of various libraries at the time of both read and write. Read performance is checked with the existing AuthorizerBenchmark. For write performance, a new AuthorizerUpdateBenchmark has been added which evaluates the performance of the addAcl operation.

        Reviewers:  Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>,  Divij Vaidya <diviv@amazon.com>

    commit 2ee770ac7e576c85a911ccb307339be3d58a8942
    Author: Colin P. McCabe <cmccabe@apache.org>
    Date:   Thu Apr 20 10:21:26 2023 -0700

        Revert "KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)"

        This reverts commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc.

    commit ef09a2e3fc11a738f6681fd57fb84ad109593fd3
    Author: Justine Olshan <jolshan@confluent.io>
    Date:   Thu Apr 20 09:30:11 2023 -0700

        KAFKA-14904: Pending state blocked verification of transactions (#13579)

        KAFKA-14561 added verification to transactional produce requests to confirm an ongoing transaction.

        There is an edge case where the transaction is added, but the coordinator is writing to the log for another partition. In this case, when verifying, we return CONCURRENT_TRANSACTIONS and retry. However, the next inflight batch is often successful because the write completes.

        When a partition has no entry in the PSM, it will allow any sequence number. This means if we retry the first write to the partition (or first write in a while) we will never be able to write it and get OutOfOrderSequence exceptions. This is a known issue. Since the verification makes this more common, I propose allowing verification on pending ongoing state since the pending state doesn't prevent us from checking the already added partitions.

        The good news is part 2 of KIP-890 will allow us to enforce that the first write for a transaction is sequence 0 and this issue will go away entirely.

        This PR also adds the locking back into the addPartitions/verify path that was incorrectly removed.

        Reviewers: Ismael Juma <ismael@juma.me.uk>, Artem Livshits <alivshits@confluent.io>, Jason Gustafson <jason@confluent.io>

    commit 7f175feacaba4187bbb3631dff3a1330060f191e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Thu Apr 20 13:50:29 2023 +0530

        KAFKA-14586: Adding redirection for StreamsResetter (#13614)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

    commit e14dd8024adae746c50c8b7d9cd268e859669576
    Author: Dimitar Dimitrov <30328539+dimitarndimitrov@users.noreply.github.com>
    Date:   Thu Apr 20 05:29:27 2023 +0200

        KAFKA-14821 Implement the listOffsets API with AdminApiDriver (#13432)

        We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic:
        1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry.
        2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of  the client.
        3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`.

        We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them.

        Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io>

    commit f5de4daa71f0ad31aa64443c5ff43c59712feea5
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Wed Apr 19 19:45:26 2023 -0400

        KAFKA-14887: FinalizedFeatureChangeListener should not shut down when ZK session expires

        FinalizedFeatureChangeListener shuts the broker down when it encounters an issue trying to process feature change
        events. However, it does not distinguish between issues related to feature changes actually failing and other
        exceptions like ZooKeeper session expiration. This introduces the possibility that Zookeeper session expiration
        could cause the broker to shutdown, which is not intended. This patch updates the code to distinguish between
        these two types of exceptions. In the case of something like a ZK session expiration it logs a warning and continues.
        We shutdown the broker only for FeatureCacheUpdateException.

        Reviewers: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Christo Lolov <christololov@gmail.com>, Colin P. McCabe <cmccabe@apache.org>

    commit 11c8bf4826197533807b2132cfc6599ba70de1c1
    Author: Victoria Xia <victoria.xia@confluent.io>
    Date:   Wed Apr 19 19:34:36 2023 -0400

        KAFKA-14834: [11/N] Update table joins to identify out-of-order records with `isLatest` (#13609)

        This PR fixes a bug in the table-table join handling of out-of-order records in versioned tables where if the latest value for a particular key is a tombstone, by using the isLatest value from the Change object instead of calling get(key) on the state store to fetch timestamps to compare against. As part of this fix, this PR also updates table-table joins to determine whether upstream tables are versioned by using the GraphNode mechanism, instead of checking the table's value getter.

        Part of KIP-914.

        Reviewer: Matthias J. Sax <matthias@confluent.io>

    commit 809966a9a06664e0b521c2298fa0de834e443607
    Author: Matthew de Detrich <matthew.dedetrich@aiven.io>
    Date:   Wed Apr 19 20:54:07 2023 +0200

        KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478)

        Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address.

        Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org>

    commit 750cfd86bf36ac7c71c5670e50eb8668f97b4246
    Author: David Arthur <mumrah@gmail.com>
    Date:   Wed Apr 19 14:19:13 2023 -0400

        KAFKA-14918 Only send controller RPCs to migrating ZK brokers (#13606)

        This patch fixes an issue where the KRaft controller could incorrectly send ZK controller RPCs to KRaft brokers.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit b10716e72370c4e128bddb17bcc107ccab221e47
    Author: hudeqi <1217150961@qq.com>
    Date:   Thu Apr 20 00:49:08 2023 +0800

        KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471)

        Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com>

    commit d04c3e56c29fc6cb876a1074b1108db2c0f37afc
    Author: Keith Wall <kwall@redhat.com>
    Date:   Wed Apr 19 03:58:29 2023 +0100

        KAFKA-14908: Set setReuseAddress on the kafka server socket (#13572)

        Changes SocketServer to set the setReuseAddress(true) socket option.

        This aids use-cases where kafka is started/stopped on the same port in rapid succession. Examples are: where a kafka cluster is embedded in an integration test suite that starts/stops a cluster before/after each test.

        Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>, Divij Vaidya <diviv@amazon.com>

    commit f905a5a45d87c94d369cde9d1326e6d18b95cf7e
    Author: vamossagar12 <sagarmeansocean@gmail.com>
    Date:   Wed Apr 19 00:46:03 2023 +0530

        MINOR: Fixing gradle build during compileScala and compileTestScala (#13588)

        Reviewers: Chia-Ping Tsai <chia7712@gmail.com>

    commit 3388adf1b52f545e2d69edd30cdd53241b2887f3
    Author: Matthias J. Sax <matthias@confluent.io>
    Date:   Tue Apr 18 11:32:27 2023 -0700

        MINOR: rename internal FK-join processor classes (#13589)

        Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>

    commit abca86511ecc9e081d676976ff6d3b845308f444
    Author: Proven Provenzano <93720617+pprovenzano@users.noreply.github.com>
    Date:   Tue Apr 18 12:41:38 2023 -0400

        KAFKA-14881: Rework UserScramCredentialRecord (#13513)

        Rework UserScramCredentialRecord to store serverKey and StoredKey rather than saltedPassword. This
        is necessary to support migration from ZK, since those are the fields we stored in ZK.  Update
        latest MetadataVersion to IBP_3_5_IV2 and make SCRAM support conditional on this version.  Moved
        ScramCredentialData.java from org.apache.kafka.image to org.apache.kafka.metadata, which seems more
        appropriate.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

    commit 61530d68ce83467de6190a52da37b3c0af84f0ef
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 09:37:04 2023 -0400

        KAFKA-14869: Bump coordinator value records to flexible versions (KIP-915, Part-2) (#13526)

        This patch implemented the second part of KIP-915. It bumps the versions of the value records used by the group coordinator and the transaction coordinator to make them flexible versions. The new versions are not used when writing to the partitions but only when reading from the partitions. This allows downgrades from future versions that will include tagged fields.

        Reviewers: David Jacot <djacot@confluent.io>

    commit b36a170aa3b2738177d7229859db765e0115385b
    Author: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
    Date:   Tue Apr 18 13:36:56 2023 +0200

        MINOR: fix typos in MigrationClient, StandardAuthorizer, StandardAuthorizerData and KafkaConfigSchema files (#13593)

        Reviewers: Luke Chen <showuon@gmail.com>

    commit 5e9d4de748dff7b91043a9b799716ab4becdae7e
    Author: Jeff Kim <kimkb2011@gmail.com>
    Date:   Tue Apr 18 04:41:54 2023 -0400

        KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, Part-1) (#13511)

        This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types.

        Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, David Jacot <djacot@confluent.io>

    commit 454b72161a76b1687a1263157d7cc30a1bdb2506
    Author: Dániel Urbán <48119872+urbandan@users.noreply.github.com>
    Date:   Tue Apr 18 09:40:14 2023 +0200

        KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows (#13557)

        KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow.
        To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread.

    commit e27926f92b1f6b34ed6731f33c712a5d0d594275
    Author: Ron Dagostino <rndgstn@gmail.com>
    Date:   Mon Apr 17 17:52:28 2023 -0400

        KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280)

        topic counts.

        Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change.  Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events.  We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods.

        Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>

    commit 7159f6c1a81ead277030f55f293943270346ad4e
    Author: Alyssa Huang <ahuang@confluent.io>
    Date:   Mon Apr 17 11:03:45 2023 -0700

        MINOR: KRaftMetadataCache.getPartitionInfo must set all relevant fields

        Fix a case where KRaftMetadataCache.getPartitionInfo was not setting all the PartitionInfo fields it
        should have been. Add a regression test.

        Reviewers: Colin P. McCabe <cmccabe@apache.org>

commit cc88822986e4004e772f376eae47dbe08e29d137
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed May 10 16:29:27 2023 -0700

    small changes

commit feb12ec2ac6e3218ee6b1e56f912da81c8d2773e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 25 14:58:20 2023 -0700

    grammar and code comments changes

commit 5550e4934ade28f069e4ae9ab97e76982ff3401f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 17:39:48 2023 -0700

    fixed formatting stuff

commit 28ab29819bf327309ce07196a3fb0492f7fb2281
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 19 11:38:57 2023 -0700

    consumers -> members

commit e56ded267456bbe1c22ded0d037953545b921d1d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 11:26:23 2023 -0700

    Interface changes incorporated

commit 2d50c278c18baa11b0235330149dd8f9def57c3d
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 13 14:47:33 2023 -0700

    Renamed topicIdToPartition class

commit d33bbfac053ccac77204ac4253698ed356d7c621
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 17:31:26 2023 -0700

    Added stickiness test

commit 0cd618affc5af5aa2f66d1cc81c2efa9cf515184
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 13:28:24 2023 -0700

    Separated uniform and general builder in different files, added more tests

commit ef814867aba68fc4813359c5cef6ae155ed49a60
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:24:27 2023 -0700

    import changes

commit dffad677c6e4a33625401d621f046af6f21feb99
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:16:33 2023 -0700

    Fixed checkstyle and added test file

commit 7fd779418df3504a490c0d10f3848360a4417077
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 5 15:13:38 2023 -0700

    First draft optimised uniform assignor code

commit 309cc6141f7962e794c90d4107f460fafd51697d
Merge: 3d9264547c 2c1cf03a89
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Mon Apr 17 11:03:07 2023 -0700

    Merge branch 'trunk' into rreddy-22/KAFKA-14514

commit 3d9264547cf9ba9afe3c27ce333bd7fe332216cc
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 10:58:16 2023 -0700

    Interface changes incorporated, added stickiness test and non-existent topic test.

commit 0737c8d2d2dce48c79a889468319c8e3bc9c9436
Merge: a1bdb58e08 57b94ca208
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 17 09:49:23 2023 -0700

    Merge remote-tracking branch 'origin/rreddy-22/KAFKA-14514' into rreddy-22/KAFKA-14514

    # Conflicts:
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/AssignmentMemberSpec.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupAssignment.java
    #	group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/MemberAssignment.java

commit a1bdb58e082d98b73f50860db5b35684f4eba51b
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 20:01:59 2023 -0700

    Removed putList, putSet, changed generic pair to remainingAssignmentsForMember, addressed PR comments, changed Map.Entry to Map.forEach etc.

commit 288fa1f5f853a0aa5d140bc930d2a8a52eeeff71
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 11:28:40 2023 -0700

    Addressed some PR comments

commit 918d262ae52892c13766649db847ede6928921f8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7369db3744b4c89f625f6b73db331eaf0512c4eb
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit 7d1626990abc6bfdf29b5c227751d10d8211ed5f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit a925b02dd3d78c8b9c1415c083f0544edcb404fe
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 9f3a423d6a2b482c6ac6d74c4e65bbed70898dfa
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit dcb8198355e82f36f12d042ed0d0a4c1d71ea8c6
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit 48a982232c6f1ff371cb44333fd12d6b37ef381a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation

commit 68d53b37b5e3fd78b8f96d7917b1fdc565e188dd
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Apr 12 13:00:07 2023 -0700

    Removed topicIdToPartition class, changed attribute names and added java doc for getter methods

commit 73c7fdcaf824a31af442b1a4aa2db81b37cc5a9e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Apr 11 15:09:44 2023 -0700

    Made all attributes private and added getter methods, changed names according to PR comments

commit 100a04e5e064be0bda392ca8f02cb7a7e89f16c8
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Fri Apr 7 11:25:33 2023 -0700

    Added toString method and hash

commit 5065109332845541c5591a52683f21d37982d80a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 13:43:26 2023 -0700

    Interface changes for assignment map and subscribed topics, added new TopicIdToPartition data structure

commit 57b94ca208cb374b22d61eb044ef8004ea09479f
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 11:12:16 2023 -0700

    Made subscribed topics a collection, removed * import exception

commit 7352cbb5683f676382f56e6ae5c926f3cae98824
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Apr 6 10:55:06 2023 -0700

    minor

commit b4041a7461c53e76d96dfa915ea7c3cd5cd9a1e0
Merge: 4b321aa537 637bc92ba1
Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com>
Date:   Thu Apr 6 10:37:58 2023 -0700

    Merge branch 'apache:trunk' into rreddy-22/KAFKA-14514

commit 4b321aa5374427a15346be9b87b7c545ccb7db61
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Apr 3 11:54:59 2023 -0700

    Addressed PR comments

commit 3b6e65ea3b8452dd923991c3dba49d370c8f0d4e
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Wed Mar 29 12:30:13 2023 -0700

    Addressed PR comments

commit 74105291cc687fe41b75de692fc624ab1335341a
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Tue Mar 28 12:21:33 2023 -0700

    removed reduce partitions case

commit 3df34605e078beb9babcbeadc7b119a70da63796
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Mon Mar 27 20:17:14 2023 -0700

    removed reduce partitions case

commit e8d36070e0b4f799bd10b6388fa1ef85e8d51328
Author: Ritika Reddy <rreddy@confluent.io>
Date:   Thu Mar 23 13:02:31 2023 -0700

    Server Side Sticky Range Assignor full implementation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mjsax mjsax mjsax left review comments

Assignees
No one assigned
Labels
kip Requires or implements a KIP streams
Projects
None yet
Milestone
No milestone
2 participants
@vcrfxia @mjsax