Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
commit 938fee2 Author: David Arthur <mumrah@gmail.com> Date: Mon Jul 31 09:21:22 2023 -0400 Fix a Scala 2.12 compile issue (apache#14126) Reviewers: Luke Chen <showuon@gmail.com>, Qichao Chu commit 3ba718e Author: Yash Mayya <yash.mayya@gmail.com> Date: Fri Jul 28 19:35:42 2023 +0100 MINOR: Remove duplicate instantiation of MockConnectMetrics in AbstractWorkerSourceTaskTest (apache#14091) Reviewers: Christo Lolov <christololov@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Greg Harris <greg.harris@aiven.io> commit 1574b9f Author: David Jacot <djacot@confluent.io> Date: Fri Jul 28 20:28:54 2023 +0200 MINOR: Code cleanups in group-coordinator module (apache#14117) This patch does a few code cleanups in the group-coordinator module. It renames Coordinator to CoordinatorShard; It renames ReplicatedGroupCoordinator to GroupCoordinatorShard. I was never really happy with this name. The new name makes more sense to me; It removes TopicPartition from the GroupMetadataManager. It was only used in log messages. The log context already includes it so we don't have to log it again. It renames assignors to consumerGroupAssignors. Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 3709901 Author: Ritika Reddy <98577846+rreddy-22@users.noreply.github.com> Date: Fri Jul 28 10:30:04 2023 -0700 KAFKA-14702: Extend server side assignor to support rack aware replica placement (apache#14099) This patch updates the `PartitionAssignor` interface to support rack-awareness. The change introduces the `SubscribedTopicDescriber` interface that can be used to retrieve topic metadata such as the number of partitions or the racks from within an assignor. We use an interface because it allows us to wrap internal data structures instead of having to copy them. It is more efficient. Reviewers: David Jacot <djacot@confluent.io> commit 32c39c8 Author: David Arthur <mumrah@gmail.com> Date: Fri Jul 28 13:02:47 2023 -0400 KAFKA-15263 Check KRaftMigrationDriver state in each event (apache#14115) Reviewers: Colin P. McCabe <cmccabe@apache.org> commit 811ae01 Author: Philip Nee <pnee@confluent.io> Date: Fri Jul 28 09:11:20 2023 -0700 MINOR: Test assign() and assignment() in the integration test (apache#14086) A missing piece from KAFKA-14950. This is to test assign() and assignment() in the integration test. Also fixed an accidental mistake in the committed API. Reviewers: Jun Rao <junrao@gmail.com> commit 19f9e1e Author: Jeff Kim <kimkb2011@gmail.com> Date: Fri Jul 28 09:13:27 2023 -0400 KAFKA-14501: Implement Heartbeat protocol in new GroupCoordinator (apache#14056) This patch implements the existing Heartbeat API in the new Group Coordinator. Reviewers: David Jacot <djacot@confluent.io> commit dcabc29 Author: David Jacot <djacot@confluent.io> Date: Fri Jul 28 14:49:48 2023 +0200 KAFKA-14048; CoordinatorContext should be protected by a lock (apache#14090) Accessing the `CoordinatorContext` in the `CoordinatorRuntime` should be protected by a lock. The runtime guarantees that the context is never access concurrently however it is accessed by multiple threads. The lock is here to ensure that we have a proper memory barrier. The patch does the following: 1) Adds a lock to `CoordinatorContext`; 2) Adds helper methods to get the context and acquire/release the lock. 3) Allow transition from Failed to Loading. Previously, the context was recreated in this case. Reviewers: Justine Olshan <jolshan@confluent.io> commit afe631c Author: James Shaw <js102@zepler.net> Date: Fri Jul 28 10:45:15 2023 +0100 KAFKA-14967: fix NPE in CreateTopicsResult in MockAdminClient (apache#13671) Co-authored-by: James Shaw <james.shaw@masabi.com> Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 722b259 Author: Christo Lolov <lolovc@amazon.com> Date: Fri Jul 28 06:40:37 2023 +0100 KAFKA-14038: Optimise calculation of size for log in remote tier (apache#14049) Reviewers: Kamal Chandraprakash<kamal.chandraprakash@gmail.com>, Divij Vaidya <diviv@amazon.com>, Luke Chen <showuon@gmail.com>, Satish Duggana <satishd@apache.org> commit 10bcd4f Author: Colin Patrick McCabe <cmccabe@apache.org> Date: Thu Jul 27 17:01:55 2023 -0700 KAFKA-15213: provide the exact offset to QuorumController.replay (apache#13643) Provide the exact record offset to QuorumController.replay() in all cases. There are several situations where this is useful, such as logging, implementing metadata transactions, or handling broker registration records. In the case where the QC is inactive, and simply replaying records, it is easy to compute the exact record offset from the batch base offset and the record index. The active QC case is more difficult. Technically, when we submit records to the Raft layer, it can choose a batch base offset later than the one we expect, if someone else is also adding records. While the QC is the only entity submitting data records, control records may be added at any time. In the current implementation, these are really only used for leadership elections. However, this could change with the addition of quorum reconfiguration or similar features. Therefore, this PR allows the QC to tell the Raft layer that a record append should fail if it would have resulted in a batch base offset other than what was expected. This in turn will trigger a controller failover. In the future, if automatically added control records become more common, we may wish to have a more sophisticated system than this simple optimistic concurrency mechanism. But for now, this will allow us to rely on the offset as correct. In order that the active QC can learn what offset to start writing at, the PR also adds a new RaftClient#endOffset function. At the Raft level, this PR adds a new exception, UnexpectedBaseOffsetException. This gets thrown when we request a base offset that doesn't match the one the Raft layer would have given us. Although this exception should cause a failover, it should not be considered a fault. This complicated the exception handling a bit and motivated splitting more of it out into the new EventHandlerExceptionInfo class. This will also let us unit test things like slf4j log messages a bit better. Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org> commit e5861ee Author: Alyssa Huang <ahuang@confluent.io> Date: Thu Jul 27 13:12:25 2023 -0700 [MINOR] Add latest versions to kraft upgrade kafkatest (apache#14084) Reviewers: Ron Dagostino <rndgstn@gmail.com> commit 6f39ef0 Author: Justine Olshan <jolshan@confluent.io> Date: Thu Jul 27 09:36:32 2023 -0700 MINOR: Adjust Invalid Record Exception for Invalid Txn State as mentioned in KIP-890 (apache#14088) Invalid record is a newer error. INVALID_TXN_STATE has been around as long as transactions and is not retriable. This is the desired behavior. commit 29825ee Author: David Jacot <djacot@confluent.io> Date: Thu Jul 27 13:18:10 2023 +0200 KAFKA-14499: [3/N] Implement OffsetCommit API (apache#14067) This patch introduces the `OffsetMetadataManager` and implements the `OffsetCommit` API for both the old rebalance protocol and the new rebalance protocol. It introduces version 9 of the API but keeps it as unstable for now. The patch adds unit tests to test the API. Integration tests will be done separately. Reviewers: Jeff Kim <jeff.kim@confluent.io>, Justine Olshan <jolshan@confluent.io> commit 353141e Author: Divij Vaidya <diviv@amazon.com> Date: Thu Jul 27 12:33:34 2023 +0200 KAFKA-15251: Add 3.5.1 to system tests (apache#14069) Reviewers: Matthias J. Sax <matthias@confluent.io> commit d2fc907 Author: Jeff Kim <kimkb2011@gmail.com> Date: Thu Jul 27 02:02:29 2023 -0400 KAFKA-14500; [6/6] Implement SyncGroup protocol in new GroupCoordinator (apache#14017) This patch implements the SyncGroup API in the new group coordinator. All the new unit tests are based on the existing scala tests. Reviewers: David Jacot <djacot@confluent.io> commit ed44bcd Author: Hao Li <1127478+lihaosky@users.noreply.github.com> Date: Wed Jul 26 16:02:52 2023 -0700 KAFKA-15022: [3/N] use graph to compute rack aware assignment for active stateful tasks (apache#14030) Part of KIP-925. Reviewers: Matthias J. Sax <matthias@confluent.io> commit 8135b6d Author: Said Boudjelda <bmscomp@gmail.com> Date: Wed Jul 26 19:52:02 2023 +0200 KAFKA-15235: Fix broken coverage reports since migration to Gradle 8.x (apache#14075) Reviewers: Divij Vaidya <diviv@amazon.com> commit e5fb9b6 Author: Said Boudjelda <bmscomp@gmail.com> Date: Wed Jul 26 19:12:27 2023 +0200 MINOR: upgrade version of gradle plugin (ben-manes.versions) to 0.47.0 (apache#14098) Reviewers: Divij Vaidya <diviv@amazon.com> commit a900794 Author: David Arthur <mumrah@gmail.com> Date: Wed Jul 26 12:54:59 2023 -0400 KAFKA-15196 Additional ZK migration metrics (apache#14028) This patch adds several metrics defined in KIP-866: * MigratingZkBrokerCount: the number of zk brokers registered with KRaft * ZkWriteDeltaTimeMs: time spent writing MetadataDelta to ZK * ZkWriteSnapshotTimeMs: time spent writing MetadataImage to ZK * Adds value 4 for "ZK" to ZkMigrationState Also fixes a typo in the metric name introduced in apache#14009 (ZKWriteBehindLag -> ZkWriteBehindLag) Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org> commit 6d81698 Author: sciclon2 <74413315+sciclon2@users.noreply.github.com> Date: Wed Jul 26 15:48:09 2023 +0200 KAFKA-15243: Set decoded user names to DescribeUserScramCredentialsResponse (apache#14094) Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> commit ff390ab Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Jul 26 17:56:20 2023 +0530 [MINOR] Fix Javadoc comment in KafkaFuture#toCompletionStage (apache#14100) Fix Javadoc comment in KafkaFuture#toCompletionStage Reviewers: Luke Chen <showuon@gmail.com> commit bb677c4 Author: Federico Valeri <fedevaleri@gmail.com> Date: Wed Jul 26 12:04:34 2023 +0200 KAFKA-14583: Move ReplicaVerificationTool to tools (apache#14059) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 4d30cbf Author: Said Boudjelda <bmscomp@gmail.com> Date: Wed Jul 26 11:21:36 2023 +0200 MINOR: Upgrade the minor version of snappy dependency to 1.1.10.3 (apache#14072) Reviewers: Divij Vaidya <diviv@amazon.com> commit 206a4af Author: Divij Vaidya <diviv@amazon.com> Date: Wed Jul 26 11:19:56 2023 +0200 MINOR: Add co-authors to release email template (apache#14080) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 46a8a28 Author: vamossagar12 <sagarmeansocean@gmail.com> Date: Wed Jul 26 07:21:23 2023 +0530 KAFKA-15218: Avoid NPE thrown while deleting topic and fetch from follower concurrently (apache#14051) When deleting topics, we'll first clear all the remoteReplicaMap when stopPartitions here. But this time, there might be fetch request coming from follower, and try to check if the replica is eligible to be added into ISR here. At this moment, NPE will be thrown. Although it's fine since this topic is already deleted, it'd be better to avoid it happen. Reviewers: Luke Chen <showuon@gmail.com> commit af1f50f Author: Matthias J. Sax <matthias@confluent.io> Date: Tue Jul 25 14:56:58 2023 -0700 MINOR: fix docs markup (apache#14085) Reviewers: Qichao Chu (@ex172000), Mickael Maison <mickael.maison@gmail.com> commit e794bc7 Author: David Arthur <mumrah@gmail.com> Date: Tue Jul 25 16:05:04 2023 -0400 MINOR: Add a Builder for KRaftMigrationDriver (apache#14062) Reviewers: Justine Olshan <jolshan@confluent.io> commit 8b027b6 Author: tison <wander4096@gmail.com> Date: Tue Jul 25 23:56:49 2023 +0800 MINOR: Fix typo in ProduceRequest.json (apache#14070) Reviewers: Mickael Maison <mickael.maison@gmail.com> commit 08b3820 Author: Yash Mayya <yash.mayya@gmail.com> Date: Tue Jul 25 14:03:29 2023 +0100 KAFKA-15238: Move DLQ reporter setup from the DistributedHerder's tick thread to the sink task thread (apache#14079) Reviewers: Chris Egerton <chrise@aiven.io> commit 58b8c5c Author: Chris Egerton <chrise@aiven.io> Date: Tue Jul 25 05:12:46 2023 -0700 MINOR: Downgrade log level for conflicting Connect plugin aliases (apache#14081) Reviewers: Greg Harris <greg.harris@aiven.io> commit c7de30f Author: Colin Patrick McCabe <cmccabe@apache.org> Date: Mon Jul 24 21:13:58 2023 -0700 KAFKA-15183: Add more controller, loader, snapshot emitter metrics (apache#14010) Implement some of the metrics from KIP-938: Add more metrics for measuring KRaft performance. Add these metrics to QuorumControllerMetrics: kafka.controller:type=KafkaController,name=TimedOutBrokerHeartbeatCount kafka.controller:type=KafkaController,name=EventQueueOperationsStartedCount kafka.controller:type=KafkaController,name=EventQueueOperationsTimedOutCount kafka.controller:type=KafkaController,name=NewActiveControllersCount Create LoaderMetrics with these new metrics: kafka.server:type=MetadataLoader,name=CurrentMetadataVersion kafka.server:type=MetadataLoader,name=HandleLoadSnapshotCount Create SnapshotEmitterMetrics with these new metrics: kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedBytes kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedAgeMs Reviewers: Ron Dagostino <rndgstn@gmail.com> commit 79b8c96 Author: David Mao <47232755+splett2@users.noreply.github.com> Date: Mon Jul 24 13:22:25 2023 -0700 KAFKA-14990: Dynamic producer ID expiration should be applied on a broker restart (apache#13707) Dynamic overrides for the producer ID expiration config are not picked up on broker restart in Zookeeper mode. Based on the integration test, this does not apply to KRaft mode. Adds a broker restart that fails without the corresponding KafkaConfig change. Reviewers: Justine Olshan <jolshan@confluent.io> commit 38781f9 Author: Justine Olshan <jolshan@confluent.io> Date: Mon Jul 24 13:08:57 2023 -0700 KAFKA-14920: Address timeouts and out of order sequences (apache#14033) When creating a verification state entry, we also store sequence and epoch. On subsequent requests, we will take the latest epoch seen and the earliest sequence seen. That way, if we try to append a sequence after the earliest seen sequence, we can block that and retry. This addresses potential OutOfOrderSequence loops caused by errors during verification (coordinator loading, timeouts, etc). Reviewers: David Jacot <david.jacot@gmail.com>, Artem Livshits <alivshits@confluent.io>
- Loading branch information
