[HUDI-8647] Support Clustering Timeline Instant Metrics in HoodieMetrics by fhan688 · Pull Request #12466 · apache/hudi

fhan688 · 2024-12-11T11:11:36Z

Change Logs

add hudi clustering timeline instant related metrics：
earliest_inflight_clustering_instant
latest_completed_clustering_instant
clustering_pendingClusteringCount

Impact

hudi-flink-datasource

Risk level (write none, low medium or high below)

Low

Documentation Update

None

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

danny0405 · 2024-12-12T02:03:44Z

hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/adapter/ContextAdapter.java

+import org.apache.flink.runtime.operators.coordination.CoordinatorStore;
+import org.apache.flink.runtime.operators.coordination.OperatorCoordinator;
+
+public class ContextAdapter implements OperatorCoordinator.Context {


So the coordinator always reference this class?

OperatorCoordinator.Context is OK for flink version 1.18、1.19、1.20， define ContextAdapter for adapting flink version 1.14、1.15、1.16、1.17.

danny0405 · 2024-12-13T02:35:14Z

...-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/metrics/FlinkTimeLineMetrics.java

+    metricGroup.gauge(PENDING_CLUSTERING_COUNT, () -> clusteringPendingClusteringCount);
+  }
+
+  public void updateTimeLineMetrics(HoodieActiveTimeline activeTimeline) {


so when the metrics is updated?

Thanks for Teacher Danny's reminding. forget to call update metrics in StreamWriteOperatorCoordinator, metrics will be updated after do commit success.

@danny0405 PTAL

danny0405 · 2024-12-16T09:34:42Z

...datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java

        reset();
        this.ckpMetadata.commitInstant(instant);
        LOG.info("Commit instant [{}] success!", instant);
+        this.flinkTimeLineMetrics.updateTimeLineMetrics(this.writeClient.getHoodieTable().getActiveTimeline());


The BaseHoodieClient already got a HoodieMetrics instance, we can just enhance it with new items or add the FlinkTimeLineMetrics in HoodieFlinkWriteClient, and update it in HoodieFlinkWriteClient.postCommit, you need to override the #postCommit method so that we can eliminate the redundant timeline loading(file listing) for each commit.

Thanks for advising, finally enhanced in HoodieMetrics . @danny0405

…ics proposal

zhangyue19921010 · 2024-12-18T09:16:15Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java

  public static final String FINALIZE_ACTION = "finalize";
  public static final String INDEX_ACTION = "index";
  public static final String SOURCE_READ_AND_INDEX_ACTION = "source_read_and_index";
+  public static final String CLUSTERTING_INSTANT_ACTION = "replacecommit";


maybe we need to take care of clustering commit instead of replace commit

zhangyue19921010 · 2024-12-18T10:10:33Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java

      // Delete the marker directory for the instant.
      WriteMarkersFactory.get(config.getMarkersType(), table, instantTime)
          .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
+      if (metrics.getClusteringInstantTimerCtx() != null) {


Check Config config.isMetricsOn() instead of trying to create a Timer.
What we want is to monitor clustering and compaction instant here rat

zhangyue19921010 · 2024-12-18T10:20:53Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java

+        Set<String> validActions = CollectionUtils.createSet(HoodieMetrics.CLUSTERTING_INSTANT_ACTION);
+        HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
+        HoodieTimeline inflightAndRequested = activeTimeline.filterInflightsAndRequested()
+                .filter(instant -> validActions.contains(instant.getAction()));
+        long pendingClusteringInstantCount = Long.valueOf(inflightAndRequested.countInstants());
+        long earliestInflightClusteringInstant = 0L;
+        Option<HoodieInstant> firstInstant = inflightAndRequested.firstInstant();
+        if (firstInstant.isPresent()) {
+          earliestInflightClusteringInstant = Long.valueOf(firstInstant.get().requestedTime());
+        }
+
+        HoodieTimeline completed = activeTimeline.filterCompletedInstants()
+            .filter(instant -> validActions.contains(instant.getAction()));
+        long latestCompletedClusteringInstant = 0L;
+        Option<HoodieInstant> lastInstant = completed.lastInstant();
+        if (lastInstant.isPresent()) {
+          latestCompletedClusteringInstant = Long.valueOf(lastInstant.get().requestedTime());
+        }


Can we move this logic into metrics.updateTimeLineClusteringInstantMetrics ? it's a little confusing that we do metrics computing in post commit here.

Thanks for advising, updated according to all above comments.

zhangyue19921010

LGTM

fhan688 · 2024-12-30T08:08:28Z

Teacher Danny, PTAL @danny0405

danny0405 · 2025-01-02T04:56:36Z

@fhan688 Can you resolve the conflicts.

zhangyue19921010 · 2025-01-03T03:04:07Z

@hudi-bot run azure

hudi-bot · 2025-01-03T04:26:50Z

CI report:

4de4d91 UNKNOWN
b1a5e59 UNKNOWN
d088adc Azure: SUCCESS Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

suport timeline metrics

4de4d91

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Dec 11, 2024

adjust ContextAdapter in hudi-flink1.14、1.15、1.16、1.17

b1a5e59

fhan688 changed the title ~~Support Timeline Metrics~~ [HUDI-8647] Support Timeline Metrics Dec 11, 2024

fix checkstyle

2c32082

danny0405 reviewed Dec 12, 2024

View reviewed changes

fhan added 2 commits December 12, 2024 12:09

rm ContextAdapter in base module and add impl in all version modules

ab8703f

fix Ut error in TestStreamWriteOperatorCoordinator

1c37c72

danny0405 reviewed Dec 13, 2024

View reviewed changes

fhan added 2 commits December 13, 2024 13:15

add call update metrics

eff5102

add register metrics

ac08efd

danny0405 reviewed Dec 16, 2024

View reviewed changes

fhan added 2 commits December 18, 2024 13:49

enhance HoodieMetrics for supporting clustering timeline related metr…

0d4b63d

…ics proposal

rm ContextAdapter related

f9ba746

github-actions bot added size:S PR with lines of changes in (10, 100] and removed size:L PR with lines of changes in (300, 1000] labels Dec 18, 2024

zhangyue19921010 reviewed Dec 18, 2024

View reviewed changes

calculate metricsin HoodieMetrics & update UT

05e0814

fhan688 changed the title ~~[HUDI-8647] Support Timeline Metrics~~ [HUDI-8647] Support Clustering Timeline Instant Metrics in HoodieMetrics Dec 18, 2024

fhan added 2 commits December 18, 2024 20:43

update compute metrics impl

b7090c3

add non-CLUSTERING_ACTION in UT

6b4eb28

zhangyue19921010 approved these changes Dec 20, 2024

View reviewed changes

danny0405 approved these changes Jan 2, 2025

View reviewed changes

Merge branch 'master' into support-timeline-metrics

d088adc

zhangyue19921010 merged commit 72d17e5 into apache:master Jan 3, 2025
42 checks passed

fhan688 mentioned this pull request Jan 21, 2025

[HUDI-8882] Support Compaction/Rollback/Clean Timeline Instant Metrics in HoodieMetrics #12681

Merged

4 tasks

hudi-bot mentioned this pull request Dec 9, 2025

Hudi Timeline Metrics Expansion #16750

Closed

Conversation

fhan688 commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyue19921010 left a comment

Choose a reason for hiding this comment

Uh oh!

fhan688 commented Dec 30, 2024

Uh oh!

danny0405 commented Jan 2, 2025

Uh oh!

zhangyue19921010 commented Jan 3, 2025

Uh oh!

hudi-bot commented Jan 3, 2025

CI report:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fhan688 commented Dec 11, 2024 •

edited

Loading