feat(common): per-partition event-time rollup; decouple watermark tracking from EVENT_TIME_ORDERING by shangxinli · Pull Request #18778 · apache/hudi

shangxinli · 2026-05-19T14:58:07Z

Describe the issue this Pull Request addresses

Part of the freshness-tracking work discussed in #17512. This PR implements Phase 1 of the reconcile plan: expose the per-partition event-time rollup that is already latent on disk, and stop gating watermark tracking on EVENT_TIME_ORDERING so freshness observability works for COW / COMMIT_TIME_ORDERING tables too.

This is purely additive — no commit-metadata key added, no avro schema change, no behavior change for tables that have not opted into hoodie.write.track.event.time.watermark.

Summary and Changelog

Today WriteStatus.markSuccess() already folds min/max event time into each HoodieWriteStat (and the avro schema already serializes them per stat alongside partitionPath). But the only public accessor on HoodieCommitMetadata is getMinAndMaxEventTime(), which collapses every partition into a single pair — consumers asking "how fresh is partition dt=2026-05-19?" have to walk partitionToWriteStats themselves.

Watermark tracking is also currently gated on recordMergeMode == EVENT_TIME_ORDERING, even though freshness observability is independent of merge semantics. The result is that COW tables with COMMIT_TIME_ORDERING silently get no watermark even when the user explicitly opts in.

This PR:

Adds HoodieCommitMetadata.getMinAndMaxEventTimePerPartition() — a pure aggregation over partitionToWriteStats that returns Map<String, Pair<Option<Long>, Option<Long>>>. Partitions whose stats carry no event time at all are omitted (so the map size reflects partitions with freshness data, not total partitions written). Min/max within a partition are folded with Math.min / Math.max, mirroring the semantics of the existing global getter. No persisted bytes, no avro change.
Decouples watermark tracking from EVENT_TIME_ORDERING in HoodieWriteHandle. Tracking now activates when eventTimeFieldName != null && hoodie.write.track.event.time.watermark=true, regardless of merge mode. The unused EVENT_TIME_ORDERING static import is removed.
Tests: five new unit tests for the rollup API in TestHoodieCommitMetadata (folding across stats within a partition, omitting partitions without event time, handling partial min/max, empty metadata, and a consistency check against the global getter); updates the existing testShouldTrackEventTimeWaterMarkerAvroRecordTypeWithCommitTimeOrdering to assert the new behavior (now tracks) and adds a negative test for the missing-event-time-field case.

Full hudi-common (1897 tests) and hudi-client-common (1026 tests) suites pass locally.

Impact

Public-API addition on HoodieCommitMetadata: external tools (catalogs, freshness exporters, lineage UIs) can now read per-partition freshness directly without walking write stats.

Behavior change for opted-in tables: COW / COMMIT_TIME_ORDERING tables with hoodie.write.track.event.time.watermark=true and an event-time field will now populate min/max on write stats; previously they were silently no-op. Tables that have not set the flag see no change.

No performance impact — the rollup is a pure in-memory aggregation that callers invoke on demand; watermark extraction at write time was already gated on the same per-record path.

Risk Level

low

The new method is additive. The behavior change is conditional on a config that is false by default and gated on an event-time field name; tables not using the flag are unaffected. Verified by running the full hudi-common and hudi-client-common test suites locally with no regressions.

Documentation Update

The hoodie.write.track.event.time.watermark config description should be updated on the Hudi website to reflect that it no longer requires EVENT_TIME_ORDERING. The new getMinAndMaxEventTimePerPartition() API is internally documented via Javadoc; a website page covering per-partition freshness consumption can land alongside Phase 2 (upstream propagation) so users see the end-to-end story in one place.

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

…tracking from EVENT_TIME_ORDERING Expose the per-partition event-time rollup that is already latent on disk and stop gating watermark tracking on EVENT_TIME_ORDERING so freshness observability works for COW / COMMIT_TIME_ORDERING tables. Changes: - HoodieCommitMetadata.getMinAndMaxEventTimePerPartition(): pure aggregation over partitionToWriteStats returning Map<String, Pair<Option<Long>, Option<Long>>>. No persisted bytes, no avro schema change. Partitions whose stats carry no event time are omitted. - HoodieWriteHandle: drop the recordMergeMode == EVENT_TIME_ORDERING check from isTrackingEventTimeWatermark. Tracking now activates whenever the event-time field is configured and hoodie.write.track.event.time.watermark is true, independent of merge mode. - Tests: 5 new TestHoodieCommitMetadata cases for the rollup API; update TestHoodieWriteHandle to assert the new merge-mode-independent behavior and add a missing-event-time-field negative case. Part of apache#17512 (Phase 1 of the reconcile plan). No behavior change for tables that have not opted into hoodie.write.track.event.time.watermark.

hudi-bot · 2026-05-19T16:39:31Z

CI report:

ed41d5a Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

codecov-commenter · 2026-05-19T17:55:24Z

Codecov Report

❌ Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.20%. Comparing base (162cac2) to head (ed41d5a).

Files with missing lines	Patch %	Lines
...apache/hudi/common/model/HoodieCommitMetadata.java	83.33%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18778      +/-   ##
============================================
+ Coverage     68.19%   68.20%   +0.01%     
- Complexity    29225    29252      +27     
============================================
  Files          2525     2525              
  Lines        141660   141677      +17     
  Branches      17591    17594       +3     
============================================
+ Hits          96607    96635      +28     
+ Misses        37097    37094       -3     
+ Partials       7956     7948       -8

Flag	Coverage Δ
common-and-other-modules	`44.35% <83.33%> (+<0.01%)`	⬆️
hadoop-mr-java-client	`44.97% <0.00%> (-0.02%)`	⬇️
spark-client-hadoop-common	`48.28% <11.11%> (-0.02%)`	⬇️
spark-java-tests	`48.86% <83.33%> (+0.03%)`	⬆️
spark-scala-tests	`44.94% <0.00%> (-0.01%)`	⬇️
utilities	`37.46% <0.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ain/java/org/apache/hudi/io/HoodieWriteHandle.java	`91.33% <ø> (-0.07%)`	⬇️
...apache/hudi/common/model/HoodieCommitMetadata.java	`86.31% <83.33%> (+1.29%)`	⬆️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hudi-agent

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the contribution! This PR adds a per-partition event-time rollup accessor on HoodieCommitMetadata and removes the EVENT_TIME_ORDERING gate from watermark tracking so COW / COMMIT_TIME_ORDERING tables can opt in. The aggregation mirrors the semantics of the existing global getter (including handling of partial min/max), is purely additive (no schema or persisted-byte change), and is well covered by the new unit tests. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. A couple of small suggestions below — mainly around extracting the duplicated fold logic into a shared helper, and tightening the Javadoc.

cc @yihua

hudi-agent · 2026-05-20T03:40:06Z

+   */
+  public Map<String, Pair<Option<Long>, Option<Long>>> getMinAndMaxEventTimePerPartition() {
+    Map<String, Pair<Option<Long>, Option<Long>>> result = new HashMap<>();
+    for (Map.Entry<String, List<HoodieWriteStat>> entry : partitionToWriteStats.entrySet()) {


🤖 nit: the sentinel-value fold (null-check → Math.min/Math.max → ternary back to Option) is essentially the same inner loop as in getMinAndMaxEventTime(). Could you extract a private foldEventTimes(List<HoodieWriteStat>) helper that both methods share? That way any future change to null-handling or sentinel semantics only needs to land in one place.

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent · 2026-05-20T03:40:06Z

+   * over its write stats, mirroring the semantics of {@link #getMinAndMaxEventTime()}.
+   *
+   * <p>This is a pure aggregation over {@code partitionToWriteStats} — it adds no persisted
+   * bytes and does not change the commit avro schema.


🤖 nit: "does not change the commit avro schema" reads more like a PR justification than an API contract — a future reader may wonder why a read-only aggregation method needs to promise schema stability, and the note could become misleading if per-partition event times are ever persisted. Have you considered dropping this sentence (or moving it to a code comment inside the method instead)?

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

danny0405

+1, reasonable

…tracking from EVENT_TIME_ORDERING (apache#18778) Expose the per-partition event-time rollup that is already latent on disk and stop gating watermark tracking on EVENT_TIME_ORDERING so freshness observability works for COW / COMMIT_TIME_ORDERING tables. Changes: - HoodieCommitMetadata.getMinAndMaxEventTimePerPartition(): pure aggregation over partitionToWriteStats returning Map<String, Pair<Option<Long>, Option<Long>>>. No persisted bytes, no avro schema change. Partitions whose stats carry no event time are omitted. - HoodieWriteHandle: drop the recordMergeMode == EVENT_TIME_ORDERING check from isTrackingEventTimeWatermark. Tracking now activates whenever the event-time field is configured and hoodie.write.track.event.time.watermark is true, independent of merge mode. - Tests: 5 new TestHoodieCommitMetadata cases for the rollup API; update TestHoodieWriteHandle to assert the new merge-mode-independent behavior and add a missing-event-time-field negative case. Part of apache#17512 (Phase 1 of the reconcile plan). No behavior change for tables that have not opted into hoodie.write.track.event.time.watermark. Co-authored-by: Xinli Shang <shangxinli@apache.org>

github-actions Bot added the size:M PR with lines of changes in (100, 300] label May 19, 2026

hudi-agent reviewed May 20, 2026

View reviewed changes

danny0405 approved these changes May 20, 2026

View reviewed changes

danny0405 merged commit 3abe066 into apache:master May 20, 2026
63 checks passed

shangxinli mentioned this pull request May 21, 2026

feat(utilities): propagate per-partition event-time watermarks from upstream Hudi sources #18802

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(common): per-partition event-time rollup; decouple watermark tracking from EVENT_TIME_ORDERING#18778

feat(common): per-partition event-time rollup; decouple watermark tracking from EVENT_TIME_ORDERING#18778
danny0405 merged 1 commit into
apache:masterfrom
shangxinli:feat/per-partition-event-time-rollup

shangxinli commented May 19, 2026

Uh oh!

hudi-bot commented May 19, 2026

Uh oh!

codecov-commenter commented May 19, 2026

Uh oh!

hudi-agent left a comment

Uh oh!

hudi-agent May 20, 2026

Uh oh!

hudi-agent May 20, 2026

Uh oh!

danny0405 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

shangxinli commented May 19, 2026

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

hudi-bot commented May 19, 2026

CI report:

Uh oh!

codecov-commenter commented May 19, 2026

Codecov Report

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 20, 2026

Choose a reason for hiding this comment

Uh oh!

hudi-agent May 20, 2026

Choose a reason for hiding this comment

Uh oh!

danny0405 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants