Dynamic configuration updates causes metrics reporting to stop

[hudi-bot]

Problem Statement:

 

When running the Hudi streamer with the [https://hudi.apache.org/releases/release-0.14.0#dynamic-configuration-updates] feature configured, the streamer restarted after a configuration change as expected; however, metrics reporting stopped.

 

I believe the sequence of events are as follows:

Hudi calls [streamSync.close()|https://code.8labs.io/third-party/github.com/apache/hudi/-/blob/release-0.14.1-scwx/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieStreamer.java?ref_type=heads#L733] which ends up [shutting down metrics reporting|https://code.8labs.io/third-party/github.com/apache/hudi/-/blob/release-0.14.1-scwx/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java?ref_type=heads#L1195]. Since the metrics instances are all [cached in a static field|https://code.8labs.io/third-party/github.com/apache/hudi/-/blob/release-0.14.1-scwx/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/Metrics.java#L47], when a new {{StreamSync}} instance is created, it creates a new HoodieMetrics instance which [reuses|https://code.8labs.io/third-party/github.com/apache/hudi/-/tree/release-0.14.1-scwx/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java#L95] the stopped {{Metrics}} instance.

 

[#10132|https://github.com//pull/10132] is a fix for a similar issue, but not related to the streamer. (That patch is already in hudi 0.14.)

 

Reproduce Steps:

 

Steps to reproduce the behavior:

• Create an implementation of the ConfigurationHotUpdateStrategy class
• Start the hudi streamer with the {{--config-hot-update-strategy-class}} option configure with your hot update strategy class
• Verify metrics reporting is working
• Change a configuration property so that your {{ConfigurationHotUpdateStrategy.updateProperties}} method returns updated properties
• Wait for the streamer to re-initialize with the updated properties
• Observe that metrics reporting no longer works

 

Expected behavior

After the streamer re-initializes metrics reporting continues to work.

 

Github Issue:

• [https://github.com/[SUPPORT] Dynamic configuration updates cause metrics reporting to stop #12282]
• [https://github.com/[SUPPORT] Dynamic configuration updates cause metrics reporting to stop #12282#issuecomment-2536237910]

JIRA info

• Link: https://issues.apache.org/jira/browse/HUDI-8778
• Type: Sub-task
• Parent: https://issues.apache.org/jira/browse/HUDI-9113
• Affects version(s):
  • 0.14.1
  • 0.15.0
• Fix version(s):
  • 1.1.0