feat(flink): add RocksDB partitioned RLI Cache Flink Options by HuangZhenQiu · Pull Request #19641 · apache/hudi

HuangZhenQiu · 2026-08-14T18:08:52Z

Exposes the 7 configuration properties defined by RFC-107 (Support data partition aware RocksDB RecordIndexBackend) as ConfigOptions in FlinkOptions

Describe the issue this Pull Request addresses

Add new configs for RFC-107 for follow up implementation.

Summary and Changelog

7 new configs are added into FlinkOptions.

Impact

none

Risk Level

none

Documentation Update

These config will be added into documents once RFC-107 is fully implemented.

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

hudi-agent

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for working on this! This PR exposes the seven RFC-107 RocksDB partitioned RLI cache configs as ConfigOptions in FlinkOptions, ahead of the actual implementation. Since these are declarations only with no wiring yet, there's little functional surface to exercise. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. A few small naming and comment nits below.

cc @yihua

hudi-agent · 2026-08-14T18:31:56Z

+
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> INDEX_RLI_ROCKSDB_CACHE_INVALIDATE_ON_REPLACE_COMMIT = ConfigOptions
+      .key("hoodie.record.index.cache.rocksdb.invalidate.on.replacecommit")


🤖 nit: replacecommit runs two words together without a separator, which breaks the standard hoodie.xxx.yyy.zzz dot-separated convention. Could you use replace.commit instead?

_{⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.}

🤖 Worth noting that replacecommit isn't an arbitrary run-together — it mirrors Hudi's own action name, HoodieTimeline.REPLACE_COMMIT_ACTION = "replacecommit" (hudi-common .../HoodieTimeline.java). Since this config keys off that action, keeping it as one token stays consistent with how the action is spelled everywhere else in the timeline; replace.commit would actually diverge from it.

🤖 One thing worth factoring in here: this key derives from the timeline action name, which is spelled as a single token — HoodieTimeline.REPLACE_COMMIT_ACTION = "replacecommit" (hudi-common/.../HoodieTimeline.java:64). Splitting it into replace.commit would actually diverge from how the action is written everywhere in the timeline, so replacecommit may be the more consistent choice despite looking run-together.

hudi-agent · 2026-08-14T18:31:56Z

+      .withDescription("Local directory for RocksDB partitioned record level index cache data.");
+
+  @AdvancedConfig
+  public static final ConfigOption<Integer> INDEX_RLI_ROCKSDB_CACHE_BOOTSTRAP_DAYS = ConfigOptions


🤖 nit: "bootstrap" is already a well-known term in Hudi (the Bootstrap feature for ingesting external tables). Using BOOTSTRAP_DAYS here for a pre-loading / warm-up window might confuse readers — have you considered PRELOAD_DAYS or WARMUP_DAYS?

_{⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent · 2026-08-14T18:31:56Z

+  public static final ConfigOption<Long> INDEX_RLI_ROCKSDB_CACHE_PARTITION_TTL_HOURS = ConfigOptions
+      .key("hoodie.record.index.cache.rocksdb.partition.ttl.hours")
+      .longType()
+      .defaultValue(168L) // default 7 days


🤖 nit: the inline // default 7 days comment duplicates what the withDescription already says — could you drop it to reduce noise?

_{⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.}

codecov-commenter · 2026-08-14T18:40:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.71%. Comparing base (558007d) to head (d2df072).

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #19641   +/-   ##
=========================================
  Coverage     77.70%   77.71%           
+ Complexity    32988    32983    -5     
=========================================
  Files          2524     2524           
  Lines        139525   139560   +35     
  Branches      16786    16786           
=========================================
+ Hits         108424   108464   +40     
+ Misses        23475    23466    -9     
- Partials       7626     7630    +4

Components	Coverage Δ
hudi-common	`83.27% <ø> (+<0.01%)`	⬆️
hudi-client	`82.73% <ø> (ø)`
hudi-flink	`85.77% <100.00%> (+0.03%)`	⬆️
hudi-spark-datasource	`71.88% <ø> (ø)`
hudi-utilities	`73.66% <ø> (-0.03%)`	⬇️
hudi-cli	`15.26% <ø> (ø)`
hudi-hadoop	`69.03% <ø> (-0.02%)`	⬇️
hudi-sync	`75.55% <ø> (ø)`
hudi-io	`79.42% <ø> (ø)`
hudi-timeline-service	`83.44% <ø> (ø)`
hudi-cloud	`64.06% <ø> (ø)`
hudi-kafka-connect	`53.96% <ø> (+0.76%)`	⬆️

Flag	Coverage Δ
common-and-other-modules	`50.89% <100.00%> (+0.01%)`	⬆️
flink-integration-tests	`49.17% <100.00%> (+0.02%)`	⬆️
hadoop-mr-java-client	`43.89% <ø> (+0.02%)`	⬆️
integration-tests	`13.63% <0.00%> (-0.01%)`	⬇️
spark-client-hadoop-common	`50.51% <ø> (-0.01%)`	⬇️
spark-java-tests	`51.72% <ø> (-0.01%)`	⬇️
spark-scala-tests	`46.49% <ø> (-0.01%)`	⬇️
utilities	`36.61% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...va/org/apache/hudi/configuration/FlinkOptions.java	`99.79% <100.00%> (+<0.01%)`	⬆️

... and 15 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hudi-agent

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for working on this! This PR adds seven RFC-107 RocksDB partitioned RLI cache configs as ConfigOptions in Flink's FlinkOptions; they're declarations only, with no consuming logic yet. No functional correctness or architectural issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review. A couple of small naming and comment nits below.

cc @yihua

hudi-agent · 2026-08-15T14:36:33Z

+
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> INDEX_RLI_ROCKSDB_CACHE_INVALIDATE_ON_REPLACE_COMMIT = ConfigOptions
+      .key("hoodie.record.index.cache.rocksdb.invalidate.on.replacecommit")


🤖 nit: replacecommit is missing a dot separator — could you rename this to hoodie.record.index.cache.rocksdb.invalidate.on.replace.commit to stay consistent with the dot-separated key convention used everywhere else?

_{⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.}

🤖 Before renaming to ...invalidate.on.replace.commit, note that this token mirrors the timeline action name, which is a single word: HoodieTimeline.REPLACE_COMMIT_ACTION = "replacecommit" (hudi-common/.../HoodieTimeline.java:64). Dotting it would diverge from the action spelling used across the timeline, so replacecommit is arguably the consistent form here.

hudi-agent · 2026-08-15T14:36:33Z

+  public static final ConfigOption<Long> INDEX_RLI_ROCKSDB_CACHE_PARTITION_TTL_HOURS = ConfigOptions
+      .key("hoodie.record.index.cache.rocksdb.partition.ttl.hours")
+      .longType()
+      .defaultValue(168L) // default 7 days


🤖 nit: the // default 7 days inline comment is already covered by the description string — could you drop it to avoid duplication that can drift if the default changes?

_{⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.}

hudi-agent · 2026-08-15T14:36:36Z

+
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> INDEX_RLI_ROCKSDB_CACHE_INVALIDATE_ON_REPLACE_COMMIT = ConfigOptions
+      .key("hoodie.record.index.cache.rocksdb.invalidate.on.replacecommit")


🤖 Worth noting that replacecommit isn't an arbitrary run-together — it mirrors Hudi's own action name, HoodieTimeline.REPLACE_COMMIT_ACTION = "replacecommit" (hudi-common .../HoodieTimeline.java). Since this config keys off that action, keeping it as one token stays consistent with how the action is spelled everywhere else in the timeline; replace.commit would actually diverge from it.

Exposes the 7 configuration properties defined by RFC-107 (Support data partition aware RocksDB RecordIndexBackend) in FlinkOptions

hudi-bot · 2026-08-15T16:42:06Z

CI report:

b32df4f UNKNOWN
d2df072 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

hudi-agent

⚠️ 🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the RFC follow-up! This exposes the seven RFC-107 RocksDB partitioned RLI cache properties as Flink ConfigOptions and aligns the RFC's config table with the index.rli.* keys. No correctness issues stand out at this stage; the remaining naming/convention points raised in earlier rounds can be settled, and a Hudi committer or PMC member can take it from here for a deeper design review.

cc @yihua

hudi-agent · 2026-08-15T18:43:17Z

+
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> INDEX_RLI_ROCKSDB_CACHE_INVALIDATE_ON_REPLACE_COMMIT = ConfigOptions
+      .key("hoodie.record.index.cache.rocksdb.invalidate.on.replacecommit")


🤖 One thing worth factoring in here: this key derives from the timeline action name, which is spelled as a single token — HoodieTimeline.REPLACE_COMMIT_ACTION = "replacecommit" (hudi-common/.../HoodieTimeline.java:64). Splitting it into replace.commit would actually diverge from how the action is written everywhere in the timeline, so replacecommit may be the more consistent choice despite looking run-together.

hudi-agent · 2026-08-15T18:43:18Z

+
+  @AdvancedConfig
+  public static final ConfigOption<Boolean> INDEX_RLI_ROCKSDB_CACHE_INVALIDATE_ON_REPLACE_COMMIT = ConfigOptions
+      .key("hoodie.record.index.cache.rocksdb.invalidate.on.replacecommit")


🤖 Before renaming to ...invalidate.on.replace.commit, note that this token mirrors the timeline action name, which is a single word: HoodieTimeline.REPLACE_COMMIT_ACTION = "replacecommit" (hudi-common/.../HoodieTimeline.java:64). Dotting it would diverge from the action spelling used across the timeline, so replacecommit is arguably the consistent form here.

HuangZhenQiu changed the title ~~feat(flink): add RFC-107 RocksDB partitioned RLI ejhgcbrbblbntdglvhjvrjbltbggddkvigjgvlhgulc options to Fli…~~ feat(flink): add RFC-107 RocksDB partitioned RLI Flink Options Aug 14, 2026

HuangZhenQiu changed the title ~~feat(flink): add RFC-107 RocksDB partitioned RLI Flink Options~~ feat(flink): add RocksDB partitioned RLI Cache Flink Options Aug 14, 2026

github-actions Bot added the size:S PR with lines of changes in (10, 100] label Aug 14, 2026

HuangZhenQiu requested a review from cshuo August 14, 2026 18:14

HuangZhenQiu force-pushed the flink-options-rfc-107 branch from 7811c79 to 4cdd217 Compare August 14, 2026 18:23

hudi-agent reviewed Aug 14, 2026

View reviewed changes

HuangZhenQiu force-pushed the flink-options-rfc-107 branch from 4cdd217 to 1948154 Compare August 15, 2026 14:28

hudi-agent reviewed Aug 15, 2026

View reviewed changes

HuangZhenQiu force-pushed the flink-options-rfc-107 branch from 1948154 to b32df4f Compare August 15, 2026 14:47

feat(flink): add RocksDB partitioned RLI Cache Flink Options

d2df072

Exposes the 7 configuration properties defined by RFC-107 (Support data partition aware RocksDB RecordIndexBackend) in FlinkOptions

HuangZhenQiu force-pushed the flink-options-rfc-107 branch from b32df4f to d2df072 Compare August 15, 2026 14:49

HuangZhenQiu requested a review from danny0405 August 15, 2026 14:49

hudi-agent reviewed Aug 15, 2026

View reviewed changes

Uh oh!

Conversation

HuangZhenQiu commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Aug 15, 2026

CI report:

Uh oh!

hudi-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HuangZhenQiu commented Aug 14, 2026 •

edited

Loading

codecov-commenter commented Aug 14, 2026 •

edited

Loading