KAFKA-9087 Replace log high watermark by future log high watermark wh… #13075

chia7712 · 2023-01-04T16:59:45Z

https://issues.apache.org/jira/browse/KAFKA-9087

[2019-06-11 17:58:46,304] ERROR [ReplicaAlterLogDirsThread-1]: Error due to (kafka.server.ReplicaAlterLogDirsThread)
 org.apache.kafka.common.KafkaException: Error processing data for partition metrics_timers-35 offset 4224887
 at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:342)
 at scala.Option.foreach(Option.scala:274)
 at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:300)
 at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:299)
 at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
 at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
 at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
 at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5(AbstractFetcherThread.scala:299)
 at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
 at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
 at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:299)
 at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:132)
 at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:131)
 at scala.Option.foreach(Option.scala:274)
 at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:131)
 at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:113)
 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
 Caused by: java.lang.IllegalStateException: Offset mismatch for the future replica metrics_timers-35: fetched offset = 4224887, log end offset = 0.
 at kafka.server.ReplicaAlterLogDirsThread.processPartitionData(ReplicaAlterLogDirsThread.scala:107)
 at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:311)

The race condition of processing LeaderAndIsrRequest and AlterReplicaLogDirsRequest causes above error (on V2 message format). Also, the error can be reproduced easily on V1 since there is no epoch cache. The ReplicaAlterLogDirsThread checks the offset of “future log” rather than “log. Hence, here is my two cents, we can replace log.highWatermark by futureLog.highWatermark to resolve this issue. I tested it on our cluster and it works well (on both V1 and V2).

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

…en adding log dir fetchers

ijuma · 2023-01-04T23:57:25Z

Is it possible to add a test?

jolshan · 2023-01-05T22:32:05Z

So the error occurs when the fetch offset is not the futureLog's log end offset.

Just curious -- is this because the log's highwatermark is lower or higher than the futureLog's?

It seems that the value we put into the InitialFetchState is the first offset we fetch from, so just wanted to clarify the race and why this fixes the issue. (Not doubting you, just trying to understand 😄 )

junrao · 2023-01-05T23:10:19Z

@jolshan : It seems that this is a regression introduced in #6841. When a future log is just created, the current log's HWM could be higher than the future replica's HWM (since it's bounded by its log end offset). When the future replica starts fetching, we want it to fetch from its local HWM, not the current log's HWM. Otherwise, there will be holes in the future log.

chia7712 · 2023-01-05T23:12:17Z

is this because the log's highwatermark is lower or higher than the futureLog's?

higher, and thanks to @junrao for the great explanation.

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala

junrao

@chia7712 : Thanks for the PR. Just one minor comment.

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala

chia7712 · 2023-01-06T20:16:23Z

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala

+    // this method should use hw of future log to create log dir fetcher. Otherwise, it causes offset mismatch error
+    val result = rm.maybeAddLogDirFetchers(Set(partition), new LazyOffsetCheckpoints(rm.highWatermarkCheckpoints), _ => None)
+    assertEquals(1, result.size)
+    assertEquals(0L, result(new TopicPartition(topic, 0)).initOffset)


@junrao Please take a look at line#258

junrao

@chia7712 : Thanks for the updated PR. A couple of more comments.

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala

junrao

@chia7712 : Thanks for the updated PR. LGTM if the tests pass.

chia7712 · 2023-01-07T07:22:30Z

unrelated error

Build / JDK 11 and Scala 2.13 / kafka.admin.ReassignPartitionsIntegrationTest.testAlterReassignmentThrottle(String).quorum=kraft | 26 秒 | 1
Build / JDK 11 and Scala 2.13 / kafka.admin.TopicCommandIntegrationTest.testTopicWithCollidingCharDeletionAndCreateAgain(String).quorum=kraft
[Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.integration.ConnectorTopicsIntegrationTest.testGetActiveTopics](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13075/7/testReport/org.apache.kafka.connect.integration/ConnectorTopicsIntegrationTest/Build___JDK_11_and_Scala_2_13___testGetActiveTopics/)

will merge it

…9-jan-2023 * apache/trunk: (16 commits) KAFKA-14570: Fix parenthesis in verifyFullFetchResponsePartitions output (apache#13072) MINOR: Remove public mutable fields from ProducerAppendInfo (apache#13091) KAFKA-14558: Move/Rewrite LastRecord, TxnMetadata, BatchMetadata, ProducerStateEntry, and ProducerAppendInfo to the storage module (apache#13043) KAFKA-14535: Fix flaky EndToEndAuthorization tests which were sensitive to ACL change reordering (apache#13086) KAFKA-9087 Replace log high watermark by future log high watermark wh… (apache#13075) MINOR: add error reason when controller failed to handle events (apache#13050) MINOR: doc: note how JDK-8136913 can affect client SASL (apache#13071) 2023 (apache#13083) KAFKA-14279; Add 3.3.x to core compatibility tests (apache#13076) MINOR Fixed doc generation for LogConfig class in genTopicConfigDocs. (apache#13079) ...

apache#13075) Reviewers: Ismael Juma <ismael@juma.me.uk>, Justine Olshan <jolshan@confluent.io>, Jun Rao <junrao@gmail.com>

KAFKA-9087 Replace log high watermark by future log high watermark wh…

3535a1a

…en adding log dir fetchers

ijuma requested a review from dajac January 4, 2023 23:57

Merge branch 'trunk' into KAFKA-9087-1

e72127f

add unit test

7fae961

chia7712 commented Jan 6, 2023

View reviewed changes

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala Outdated Show resolved Hide resolved

jolshan reviewed Jan 6, 2023

View reviewed changes

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala Outdated Show resolved Hide resolved

chia7712 added 3 commits January 7, 2023 00:58

Merge branch 'trunk' into KAFKA-9087-1

69c1fe1

address comment

1139100

revert

d330da3

junrao reviewed Jan 6, 2023

View reviewed changes

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala Show resolved Hide resolved

check offset

9a0d014

chia7712 commented Jan 6, 2023

View reviewed changes

junrao reviewed Jan 6, 2023

View reviewed changes

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala Outdated Show resolved Hide resolved

core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala Show resolved Hide resolved

Address comment

79343b7

junrao approved these changes Jan 6, 2023

View reviewed changes

chia7712 merged commit b99be76 into apache:trunk Jan 7, 2023

chia7712 mentioned this pull request Jan 7, 2023

Data Directory 的搬移導致 Kafka Broker 內部的錯誤 skiptests/astraea#1325

Closed

chia7712 deleted the KAFKA-9087-1 branch March 25, 2024 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-9087 Replace log high watermark by future log high watermark wh… #13075

KAFKA-9087 Replace log high watermark by future log high watermark wh… #13075

chia7712 commented Jan 4, 2023 •

edited

ijuma commented Jan 4, 2023

jolshan commented Jan 5, 2023

junrao commented Jan 5, 2023

chia7712 commented Jan 5, 2023

junrao left a comment

chia7712 Jan 6, 2023

junrao left a comment

junrao left a comment

chia7712 commented Jan 7, 2023

KAFKA-9087 Replace log high watermark by future log high watermark wh… #13075

KAFKA-9087 Replace log high watermark by future log high watermark wh… #13075

Conversation

chia7712 commented Jan 4, 2023 • edited

Committer Checklist (excluded from commit message)

ijuma commented Jan 4, 2023

jolshan commented Jan 5, 2023

junrao commented Jan 5, 2023

chia7712 commented Jan 5, 2023

junrao left a comment

Choose a reason for hiding this comment

chia7712 Jan 6, 2023

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

chia7712 commented Jan 7, 2023

chia7712 commented Jan 4, 2023 •

edited