feat: Add configurable cleaner policy for metadata table by suryaprasanna · Pull Request #17935 · apache/hudi

suryaprasanna · 2026-01-18T03:34:28Z

Describe the issue this Pull Request addresses

Metadata table currently inherits the cleaning policy from the data table, which may not always be optimal for metadata table operations. This PR introduces a dedicated configurable cleaning policy for the metadata table that can either use main table's clean or use KEEP_LATEST_FILE_VERSIONS which is recommended strategy for MOR table type.

Summary and Changelog

This PR adds support for configuring the metadata table's cleaning policy independently from the data table. Users can now set hoodie.metadata.clean.policy to control how the metadata table performs cleaning operations.

Changes:

Added new config hoodie.metadata.use.main.table.clean.policy in HoodieMetadataConfig with default value as true
Modified HoodieMetadataWriteUtils.createMetadataWriteConfig() to use either KEEP_LATEST_FILE_VERSIONS stratgy for clean or to inherit clean policy from data table.

Impact

Users can now independently configure metadata table cleaning behavior. Creating a way for metadata clean policy to use (KEEP_LATEST_FILE_VERSIONS) as it ensures efficient file management regardless of the data table's cleaning strategy.

Risk Level

low - This change is backward compatible. The hoodie.metadata.use.main.table.clean.policy config makes sure the original logic is not overridden.

Documentation Update

New config:

hoodie.metadata.use.main.table.clean.policy (advanced): Determines the cleaner policy for metadata table. Default: true.

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

nsivabalan · 2026-01-23T08:54:00Z

...ient/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java

+        .withCleanerPolicy(metadataTableCleaningPolicy);

-    if (HoodieCleaningPolicy.KEEP_LATEST_COMMITS.equals(dataTableCleaningPolicy)) {
+    if (HoodieCleaningPolicy.KEEP_LATEST_COMMITS.equals(metadataTableCleaningPolicy)) {


I am not sure if 1.2 X is needed anymore.
previously, there is no way one can configure cleaner configuration for mdt, and hence we were deriving from data table's clean policy and hence added the 20% buffer.
but if we are looking to add explicit configurations, we don't need buffer.

nsivabalan · 2026-01-23T08:57:21Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

+      .defaultValue(KEEP_LATEST_FILE_VERSIONS.name())
+      .markAdvanced()
+      .sinceVersion("1.2.0")
+      .withDocumentation("This config determines the cleaner policy for metadata table.");


can we also add a boolean Config property named hoodie.metadata.clean.follow.data.table.policy and enable it by default.

out of the box, lets fall back to old behavior with 1.2X multiplier.

but advanced users will have a choice to override this and use hoodie.metadata.clean.policy on a need basis.
I am worried that, some data table could have 7 day cleaner retention, but switching to this FILE versions based and not deriving the value from data table might have an impact

oh, here we are only adding the policy override. the values required for these policies are still derived from data table.
but are we not making a assumption here that, both data table and metadata will be using same clean policy.

what incase users configure different policy for each data table and mdt table.

may be, if the policy does not align between data table and mdt, we could throw exceptions.
and also add a configuration for the multiplier. leave the default as 1.2 x

I feel adding more configs will unnecessarily complicate the code. Let us just maintain one config for metadata cleaner policy, we can decide which one it should be.

nsivabalan · 2026-01-23T08:58:26Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

+      .defaultValue(KEEP_LATEST_FILE_VERSIONS.name())
+      .markAdvanced()
+      .sinceVersion("1.2.0")
+      .withDocumentation("This config determines the cleaner policy for metadata table.");


oh, here we are only adding the policy override. the values required for these policies are still derived from data table.
but are we not making a assumption here that, both data table and metadata will be using same clean policy.

what incase users configure different policy for each data table and mdt table.

nsivabalan · 2026-01-23T09:00:14Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

+      .defaultValue(KEEP_LATEST_FILE_VERSIONS.name())
+      .markAdvanced()
+      .sinceVersion("1.2.0")
+      .withDocumentation("This config determines the cleaner policy for metadata table.");


may be, if the policy does not align between data table and mdt, we could throw exceptions.
and also add a configuration for the multiplier. leave the default as 1.2 x

nsivabalan · 2026-01-23T09:00:42Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

      return this;
    }

+    public HoodieMetadataConfig.Builder withCleanerPolicy(HoodieCleaningPolicy policy) {


We need to add tests.
check out TestHoodieMetadataConfig

nsivabalan · 2026-01-27T22:10:38Z

...ient/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java


    final long maxLogFileSizeBytes = writeConfig.getMetadataConfig().getMaxLogFileSize();
    // Borrow the cleaner policy from the main table and adjust the cleaner policy based on the main table's cleaner policy
+    boolean useMainTableCleanPolicy = writeConfig.getMetadataConfig().useMainTableCleanPolicy();


driveFromDatatableCleanPolicy

nsivabalan · 2026-01-27T22:11:22Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

      .withDocumentation("Controls the criteria to log compacted files groups in metadata table.");

+  public static final ConfigProperty<Boolean> USE_MAIN_TABLE_CLEAN_POLICY = ConfigProperty
+      .key(METADATA_PREFIX + ".use.main.table.clean.policy")


METADATA_PREFIX + derive.datatable.clean.policy

nsivabalan · 2026-01-27T22:11:39Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

+      .defaultValue(true)
+      .markAdvanced()
+      .sinceVersion("1.2.0")
+      .withDocumentation("This config determines whether the cleaner policy should use main table's cleaner policy.");


fix the wording. main -> data

nsivabalan · 2026-01-27T22:11:56Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

    return getIntOrDefault(RECORD_PREPARATION_PARALLELISM);
  }

+  public boolean useMainTableCleanPolicy() {


fix the naming once the config key and property is fixed

nsivabalan · 2026-01-27T22:12:03Z

hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java

      return this;
    }

+    public HoodieMetadataConfig.Builder withMainTableCleanPolicy(boolean useMainTableCleanPolicy) {


nsivabalan · 2026-01-27T22:14:26Z

LGTM. mostly minor comments

…a table

hudi-bot · 2026-01-28T01:57:33Z

CI report:

24e82ff Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

github-actions bot added the size:S PR with lines of changes in (10, 100] label Jan 18, 2026

nsivabalan reviewed Jan 23, 2026

View reviewed changes

suryaprasanna force-pushed the surya-dev-04 branch from 7631ef2 to e427f02 Compare January 26, 2026 22:07

github-actions bot added size:M PR with lines of changes in (100, 300] and removed size:S PR with lines of changes in (10, 100] labels Jan 26, 2026

nsivabalan reviewed Jan 27, 2026

View reviewed changes

suryaprasanna added 6 commits January 27, 2026 22:46

[UBER] Change cleaner policy to KEEP_LATEST_FILE_VERSIONS for metadat…

c1a057c

…a table

Fix compilation issues

47f62ec

Make code changes

24ac4f9

WIP

e891daa

Create unit test

98fe077

Address review comments

24e82ff

suryaprasanna force-pushed the surya-dev-04 branch from e427f02 to 24e82ff Compare January 27, 2026 22:47

nsivabalan approved these changes Jan 28, 2026

View reviewed changes

nsivabalan merged commit 1ed2821 into apache:master Jan 28, 2026
72 checks passed

alexr17 pushed a commit to alexr17/hudi that referenced this pull request Jan 30, 2026

feat: Add configurable cleaner policy for metadata table (apache#17935)

06e869e

PavithranRick pushed a commit to PavithranRick/hudi that referenced this pull request Feb 2, 2026

feat: Add configurable cleaner policy for metadata table (apache#17935)

e2b0864

alexr17 pushed a commit to alexr17/hudi that referenced this pull request Feb 5, 2026

feat: Add configurable cleaner policy for metadata table (apache#17935)

f3d9f20

prashantwason pushed a commit to prashantwason/incubator-hudi that referenced this pull request Feb 23, 2026

feat: Add configurable cleaner policy for metadata table (apache#17935)

10e5e43

Conversation

suryaprasanna commented Jan 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan commented Jan 27, 2026

Uh oh!

hudi-bot commented Jan 28, 2026

CI report:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

suryaprasanna commented Jan 18, 2026 •

edited

Loading