[HUDI-9340] Adding streaming support to secondary index writes w/ metadata table #13436

nsivabalan · 2025-06-14T16:32:42Z

Change Logs

To be filled in.

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

nsivabalan · 2025-06-16T00:23:44Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/IndexStats.java

+  }
+
+  public void addSecondaryIndexStats(String secondaryIndexPartitionPath, String recordKey, String secondaryIndexValue, boolean isDeleted) {
+    secondaryIndexStats.computeIfAbsent(secondaryIndexPartitionPath, k -> new ArrayList<>())


can you add an explicit method named initializeSecondaryIndexStats.
and within constructor of HoodieWriteHandle, we can call this method when we know sec index is enabled and if there are any sec index defns found.

nsivabalan · 2025-06-16T00:24:08Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/IndexStats.java

+  }
+
+  void clear() {
+    this.writtenRecordDelegates.clear();


can you clear secondaryIndexStats as well

nsivabalan · 2025-06-16T00:24:15Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/SecondaryIndexStats.java

+ */
+public class SecondaryIndexStats {
+
+  private String recordKey;


nsivabalan · 2025-06-16T00:26:25Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java

@@ -62,7 +65,7 @@ public class HoodieCreateHandle<T, I, K, O> extends HoodieWriteHandle<T, I, K, O
  protected long recordsDeleted = 0;
  private Map<String, HoodieRecord<T>> recordMap;
  private boolean useWriterSchema = false;
-  private final boolean preserveMetadata;
+  private boolean preserveMetadata;


why reverting this ?

Removed this change. I had addded it for fixing one of the tests. Not required anymore.

nsivabalan · 2025-06-16T00:26:54Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java

@@ -173,6 +178,23 @@ record = record.prependMetaFields(schema, writeSchemaWithMetaFields, new Metadat
    }
  }

+  private void trackMetadataIndexStats(HoodieRecord record) {
+    if (!config.isSecondaryIndexEnabled() || secondaryIndexDefns.isEmpty() || !config.isMetadataStreamingWritesEnabled(hoodieTable.getMetaClient().getTableConfig().getTableVersion())) {


can you move this condition to HoodieWriteHandle and use it all sub class.

nsivabalan · 2025-06-16T00:38:05Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java

@@ -452,6 +458,51 @@ protected HoodieRecord<T> updateFileName(HoodieRecord<T> record, Schema schema,
    return record.prependMetaFields(schema, targetSchema, metadataValues, prop);
  }

+  private void trackMetadataIndexStats(Option<HoodieKey> hoodieKeyOpt, Option<HoodieRecord> combinedRecordOpt, Option<HoodieRecord<T>> oldRecordOpt, boolean isDelete) {
+    if (!config.isSecondaryIndexEnabled() || secondaryIndexDefns.isEmpty() || !config.isMetadataStreamingWritesEnabled(hoodieTable.getMetaClient().getTableConfig().getTableVersion())) {


based on earlier feedback, we should be able to access a protected method from HoodieWriteHandle for this

nsivabalan · 2025-06-16T00:41:53Z

...di-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java

@@ -149,7 +152,8 @@ public abstract class HoodieBackedTableMetadataWriter<I, O> implements HoodieTab
  private static final String RECORD_KEY_FIELD_NAME = HoodieMetadataPayload.KEY_FIELD_NAME;

  // tracks the list of MDT partitions which can write to metadata table in a streaming manner.
-  private static final List<MetadataPartitionType> STREAMING_WRITES_SUPPORTED_PARTITIONS = Arrays.asList(RECORD_INDEX);
+  private static final List<MetadataPartitionType> STREAMING_WRITES_SUPPORTED_PARTITIONS = Arrays.asList(RECORD_INDEX, SECONDARY_INDEX);
+  private static final List<MetadataPartitionType> STREAMING_WRITES_SUPPORTED_PARTITION_PREFIXES = Arrays.asList(SECONDARY_INDEX);


Note to Reviewer:
Certain partitions in MDT are one on one mapped to the MetadataPartitionType. but its not always the case.
For eg, in case of secondary index, we generally do startsWtith and not really exact match, since we could have N no of sec indexes built.

can we enhance tests that we wrote for the patch where we added APIs and impl for the streaming metadata writes.

nsivabalan · 2025-06-16T00:45:45Z

hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestAppendHandle.java

+      assertEquals(1, writeStatus.getIndexStats().getSecondaryIndexStats().size());
+      // Since the MDT is not populated during the create, the updates would be considered as new records by the Append handle
+      // Therefore only secondary index records for the 50 updates would appear here
+      assertEquals(50, writeStatus.getIndexStats().getSecondaryIndexStats().values().stream().findFirst().get().size());


can we validate the content as well

nsivabalan · 2025-06-16T00:47:20Z

hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestCreateHandle.java

+    // validate write status has all record delegates
+    if (populateMetaFields) {
+      assertEquals(1, writeStatus.getIndexStats().getSecondaryIndexStats().size());
+      assertEquals(100, writeStatus.getIndexStats().getSecondaryIndexStats().values().stream().findFirst().get().size());


nsivabalan · 2025-06-16T00:47:59Z

hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/io/TestMergeHandle.java

+    // verify secondary index stats
+    assertEquals(1, writeStatus.getIndexStats().getSecondaryIndexStats().size());
+    // 10 si records for old secondary keys and 10 for new secondary keys
+    assertEquals(20, writeStatus.getIndexStats().getSecondaryIndexStats().values().stream().findFirst().get().size());


we should def validate the contents here.

lets ensure there is a mix of updates and deletes as well.

nsivabalan · 2025-06-16T15:44:53Z

btw, do we have tests for FG reader based Merge Handle as well?
compaction takes that route right.

nsivabalan · 2025-06-17T06:24:28Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java

+  }
+
+  /**
+   * Returns true if streaming is enabled and secondary index is configured for the table.


fix java docs

nsivabalan · 2025-06-17T06:30:48Z

we need to ignore SI record generation for compaction and clustering.

hudi-bot · 2025-06-17T07:05:59Z

CI report:

9c2caf5 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

github-actions bot added the size:XL label Jun 14, 2025

nsivabalan commented Jun 16, 2025

View reviewed changes

nsivabalan and others added 12 commits June 17, 2025 10:50

Adding secondary index stats to Write Status

3982135

temp changes to support streaming secondary index stats generation

7013528

Fixes for create and merge handle

c84da80

Fixing HoodieMergeHandle secondary index stats collection

79645b3

Add si support in Append handle

d797211

Refactoring changes

2b22bbd

Fix test

f0d1ac1

Adding streaming support to secondary index with metadata

54dc8fc

Reverting unnecessary change

c899b7a

minor fixes to write handles

b799c1c

Address review comments

9cf6c3c

Fix test failure

9c2caf5

lokeshj1703 force-pushed the HUDI-9417-HUDI-9405-end-to-end-stitching-collapsed-si-streaming branch from 73887b7 to 9c2caf5 Compare June 17, 2025 05:49

nsivabalan changed the title ~~[Hudi 9340][DNM] Adding streaming support to secondary index writes w/ metadata table~~ [HUDI-9340] Adding streaming support to secondary index writes w/ metadata table Jun 17, 2025

nsivabalan commented Jun 17, 2025

View reviewed changes

nsivabalan closed this Jun 25, 2025

[HUDI-9340] Adding streaming support to secondary index writes w/ metadata table #13436

[HUDI-9340] Adding streaming support to secondary index writes w/ metadata table #13436

Uh oh!

Conversation

nsivabalan commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan commented Jun 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan commented Jun 17, 2025

Uh oh!

hudi-bot commented Jun 17, 2025

CI report:

Uh oh!

Uh oh!

nsivabalan commented Jun 14, 2025 •

edited

Loading