chore(flink): cherrypick append write code path fixes by HuangZhenQiu · Pull Request #18757 · apache/hudi

HuangZhenQiu · 2026-05-16T14:07:07Z

Describe the issue this Pull Request addresses

Cherrypick Append write code path fixes

Summary and Changelog

#7611
#9357
#9867
#10119
#10767
#10312

Impact

none

Risk Level

none

Documentation Update

none

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

…ial uncommitted write metadata event (apache#7611)

…ing heartbeat for lazy failed writes clean policy (apache#9135)

…delete heartbeat files for close (apache#9160)

…pache#9357) Co-authored-by: danny0405 <yuzhao.cyz@gmail.com>

…pache#10119)

… handle CheckPointComplete not finished (apache#10767) Test Plan: unit tests Reviewers: O955 Project Hoodie Project Reviewer: Add blocking reviewers, #hoodie_blocking_reviewers, peter.huang, vamshi Reviewed By: O955 Project Hoodie Project Reviewer: Add blocking reviewers, #hoodie_blocking_reviewers, peter.huang, vamshi Tags: #has_java JIRA Issues: HUDI-6847 Differential Revision: https://code.uberinternal.com/D17898557

hudi-bot · 2026-05-16T16:01:51Z

CI report:

bc2b16a Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

hudi-agent

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the cherry-pick! This brings several Append-write-path fixes from master into release-0.14.2. The main piece I'd ask you to double-check is the ConcurrentHashMap → HashMap swap in HoodieHeartbeatClient, since the internal Timer thread reads/writes the map concurrently with the main thread. A couple of other questions in the inline comments. Please take a look at any inline comments, and this should be ready for a Hudi committer or PMC member to take it from here. One naming/import nit worth addressing; the rest of the changes look clean.

hudi-agent · 2026-05-16T17:14:06Z

    this.heartbeatIntervalInMs = heartbeatIntervalInMs;
    this.maxAllowableHeartbeatIntervalInMs = this.heartbeatIntervalInMs * numTolerableHeartbeatMisses;
-    this.instantToHeartbeatMap = new ConcurrentHashMap<>();
+    this.instantToHeartbeatMap = new HashMap<>();


🤖 I'm a bit worried about downgrading from ConcurrentHashMap to HashMap here. The HeartbeatTask runs on the internal Timer thread (Timer(true) in Heartbeat) and reads instantToHeartbeatMap via updateHeartbeat while the caller thread can be doing put/remove/clear (e.g. start, stop, close). Even though the class is @NotThreadSafe for external callers, the internal Timer thread isn't external — race between Timer thread get and main thread put/clear on a plain HashMap can corrupt the map state. Was there a specific reason for the change? @danny0405

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent · 2026-05-16T17:14:06Z

+        return;
+      }
+
      initInstant(instant);


🤖 Just to double-check the scope: this reuse path only triggers when tableState.operationType == INSERT. For other append-style ops (e.g. INSERT_OVERWRITE, BULK_INSERT) the bootstrap will still re-init a new instant and the prior pending one will be rolled back — is that intentional, or should this also cover those operation types where the same "data was snapshotted but not committed" scenario can occur?

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent · 2026-05-16T17:14:06Z

-    if (writeClient.getConfig().getFailedWritesCleanPolicy().isLazy() && !WriteMetadataEvent.BOOTSTRAP_INSTANT.equals(this.instant)) {
-      writeClient.getHeartbeatClient().stop(this.instant);
-    }
    // starts a new instant


🤖 With the explicit heartbeatClient.stop(this.instant) removed from initInstant, the only path that stops the prior instant's heartbeat is writeClient.commit(...) inside the recommit branch. If initInstant takes the reset()/no-recommit path (e.g. the instant was already in the completed timeline but a prior pending this.instant still has a live heartbeat, or the bootstrap path resets without recommit), does the prior heartbeat get cleaned up anywhere, or could it leak until process exit? Worth confirming there's no scenario where a stale heartbeat is left running for the previous this.instant.

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent · 2026-05-16T17:14:06Z

-import org.apache.hudi.util.ClusteringUtil;
-import org.apache.hudi.util.CompactionUtil;
-import org.apache.hudi.util.FlinkWriteClients;
+import org.apache.hudi.util.*;


🤖 nit: wildcard imports are generally discouraged in Java (and flagged by most checkstyle configs) — could you restore the four explicit imports (ClientIds, ClusteringUtil, CompactionUtil, FlinkWriteClients) rather than using org.apache.hudi.util.*?

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for cherry-picking these fixes! This PR backports a set of upstream append-write fixes (heartbeat handling, bootstrap event reuse on partial failover, and related test helpers). Prior review rounds have already flagged the key concerns (HashMap thread-safety, heartbeat lifecycle on initInstant, INSERT-only scope, wildcard imports); no new correctness or architectural issues surfaced from this pass. Please take a look at the existing inline comments, and this should be ready for a Hudi committer or PMC member to take it from here. A couple of minor readability nits below — one on the log format placeholder and one on the Javadoc wording; the rest of the changes look clean.

cc @yihua

hudi-agent · 2026-05-16T18:45:56Z

+      if (this.metaClient.reloadActiveTimeline().filterInflightsAndRequested().containsInstant(this.instant)
+              && instant.equals(WriteMetadataEvent.BOOTSTRAP_INSTANT)
+              && this.tableState.operationType == WriteOperationType.INSERT) {
+        LOG.warn("Reuse current pending Instant {} with {} operationType, "


🤖 nit: the {} placeholder is always filled with WriteOperationType.INSERT.value() since we just checked operationType == INSERT — it could be confusing to a reader who wonders why it's a placeholder if it never varies. Could you either inline "INSERT" in the string, or use this.tableState.operationType.value() so it reads as a runtime value like the first {}?

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent · 2026-05-16T18:45:57Z

  }

+  /**
+   * Triggers Job level fail, so the coordinator need re-create a new instance.


🤖 nit: "Triggers Job level fail" describes the scenario rather than what the method does, and it blurs the distinction with coordinatorFails() which already exists. Something like "Simulates a coordinator restart by closing and re-creating the coordinator instance" would be clearer — and "need re-create" should be "needs to re-create".

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

danny0405 · 2026-05-18T02:07:32Z

    this.heartbeatIntervalInMs = heartbeatIntervalInMs;
    this.maxAllowableHeartbeatIntervalInMs = this.heartbeatIntervalInMs * numTolerableHeartbeatMisses;
-    this.instantToHeartbeatMap = new ConcurrentHashMap<>();
+    this.instantToHeartbeatMap = new HashMap<>();


why this change

SteNicholas and others added 6 commits May 16, 2026 06:50

[HUDI-5506] StreamWriteOperatorCoordinator may not recommit with part…

1955c96

…ial uncommitted write metadata event (apache#7611)

[HUDI-6501] StreamWriteOperatorCoordinator should recommit with start…

3b5df9b

…ing heartbeat for lazy failed writes clean policy (apache#9135)

[HUDI-6501] HoodieHeartbeatClient should stop all heartbeats and not …

3c7b2f0

…delete heartbeat files for close (apache#9160)

[HUDI-6588] Fix duplicate fileId on TM partial-failover and recovery (a…

98568c9

…pache#9357) Co-authored-by: danny0405 <yuzhao.cyz@gmail.com>

[HUDI-6944] Fix flink boostrap concurrency issue (apache#9867)

f2973da

[HUDI-7109] Fix Flink may re-use a committed instant in append mode (a…

ccf69bc

…pache#10119)

github-actions Bot added the size:M PR with lines of changes in (100, 300] label May 16, 2026

HuangZhenQiu force-pushed the cherry-pick-fix1 branch from 67f0ffd to bc2b16a Compare May 16, 2026 14:34

HuangZhenQiu changed the title ~~Cherrypick Append write code path fixes~~ chore(flink): cherrypick append write code path fixes May 16, 2026

github-actions Bot added size:S PR with lines of changes in (10, 100] and removed size:M PR with lines of changes in (100, 300] labels May 16, 2026

hudi-agent reviewed May 16, 2026

View reviewed changes

danny0405 reviewed May 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(flink): cherrypick append write code path fixes#18757

chore(flink): cherrypick append write code path fixes#18757
HuangZhenQiu wants to merge 7 commits into
apache:release-0.14.2from
HuangZhenQiu:cherry-pick-fix1

HuangZhenQiu commented May 16, 2026

Uh oh!

hudi-bot commented May 16, 2026

Uh oh!

hudi-agent left a comment

Uh oh!

hudi-agent May 16, 2026

Uh oh!

hudi-agent May 16, 2026

Uh oh!

hudi-agent May 16, 2026

Uh oh!

hudi-agent May 16, 2026

Uh oh!

hudi-agent left a comment

Uh oh!

hudi-agent May 16, 2026

Uh oh!

hudi-agent May 16, 2026

Uh oh!

danny0405 May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

HuangZhenQiu commented May 16, 2026

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

hudi-bot commented May 16, 2026

CI report:

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 16, 2026

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 16, 2026

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 16, 2026

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 16, 2026

Choose a reason for hiding this comment

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 16, 2026

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 16, 2026

Choose a reason for hiding this comment

Uh oh!

danny0405 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants