[HUDI-6801] Implement merging partial updates from log files for MOR tables by yihua · Pull Request #9883 · apache/hudi

yihua · 2023-10-19T01:50:27Z

Change Logs

This PR adds the logic of merging partial updates in the new file group reader with Spark record merger.

Impact

Supports snapshot queries on file groups with partial updates.

Risk level

low

Documentation Update

N/A

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecord.java

...-common/src/main/java/org/apache/hudi/common/table/read/HoodieBaseFileGroupRecordBuffer.java

...ient/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java

danny0405 · 2023-10-26T03:27:09Z

hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java

    Map<String, Object> meta = new HashMap<>();
    meta.put(INTERNAL_META_RECORD_KEY, getRecordKey(record, schema));
+    meta.put(INTERNAL_META_SCHEMA, schema);
+    meta.put(INTERNAL_META_IS_PARTIAL, isPartial);


I'm wondering whether we can represent the metadata as a POJO to make the interface more explicit and clear.

We can. We can take it up in a separate PR.

hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java

danny0405 · 2023-10-26T03:30:50Z

...mon/src/main/java/org/apache/hudi/common/table/read/HoodieKeyBasedFileGroupRecordBuffer.java

      String recordKey = readerContext.getRecordKey(baseRecord, baseFileSchema);
      Pair<Option<T>, Map<String, Object>> logRecordInfo = records.remove(recordKey);
+      Map<String, Object> metadata = readerContext.generateMetadataForRecord(
+          baseRecord, baseFileSchema, false);


Caution for the performace regression for per-record metadata construction.

We'll do benchmarking on this. Likely it's ok given we're not doing much work inside the method and the existing log reader also extract some metadata for each record.

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java

...di-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala

…tables

…tial updates

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java

…s only

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java

danny0405 · 2023-10-28T03:57:23Z

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java

    boolean isValid = data == null || data instanceof UnsafeRow
-        || schema != null && (data instanceof HoodieInternalRow || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data));
+        || schema != null && (data instanceof HoodieInternalRow
+        || data instanceof GenericInternalRow


The original indentation is more clear.

The line is too long. I've revised the indentation to make it more clear.

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java

...rc/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedFileGroupRecordBuffer.java

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java

danny0405

+1, let's supplement with more tests if it is necessary.

…Reader to read parquet log blocks

…tables (apache#9883) This commit adds the logic of merging partial updates in the new file group reader with Spark record merger.

yihua force-pushed the HUDI-6801-merging-partial-updates branch from 7eb16f5 to 2de35c7 Compare October 19, 2023 05:26

yihua added the release-1.0.0 label Oct 20, 2023

yihua changed the title ~~[HUDI-6801][WIP] Implement merging partial updates from log files for MOR …~~ [HUDI-6801] Implement merging partial updates from log files for MOR tables Oct 21, 2023

yihua force-pushed the HUDI-6801-merging-partial-updates branch 5 times, most recently from a54b000 to c140ff4 Compare October 23, 2023 06:27

yihua marked this pull request as ready for review October 23, 2023 06:27

danny0405 requested changes Oct 23, 2023

View reviewed changes

yihua force-pushed the HUDI-6801-merging-partial-updates branch 2 times, most recently from 085a858 to 0b36862 Compare October 25, 2023 23:28

danny0405 reviewed Oct 26, 2023

View reviewed changes

yihua added 10 commits October 25, 2023 23:53

[HUDI-6801] Implement merging partial updates from log files for MOR …

423dcd4

…tables

Fix merging partial updates based on write-side changes

ce86903

Fix merging after rebasing

b6019b2

Remove isPartial flag in HoodieRecord and use schema to determine par…

9ab8486

…tial updates

Remove isLog argument

e8a4b0e

Add test on validating merging results

6908587

Relax checkstyle and fix style issues

97ff7dc

Remove local tests

e3e0718

Address review comments

2e4efae

Fix reading parquet log block with Spark vectorized reader

48be011

yihua force-pushed the HUDI-6801-merging-partial-updates branch from 78480b5 to 48be011 Compare October 26, 2023 07:02

Address review comments and code style issues

10f2ad7

danny0405 reviewed Oct 26, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java Outdated Show resolved Hide resolved

danny0405 reviewed Oct 26, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java Outdated Show resolved Hide resolved

danny0405 reviewed Oct 26, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java Outdated Show resolved Hide resolved

danny0405 reviewed Oct 26, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java Outdated Show resolved Hide resolved

danny0405 reviewed Oct 26, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java Show resolved Hide resolved

yihua added 6 commits October 26, 2023 13:42

Fix parquet data block reading

c8f6720

Address review comment

5b39c6d

Fix test

8d820be

WIP

169971b

Fix log file check

a54a35d

Add partial merging API and enable partial merging for partial update…

03c7595

…s only

danny0405 reviewed Oct 28, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/HoodieSparkRecordMerger.java Show resolved Hide resolved

danny0405 reviewed Oct 28, 2023

View reviewed changes

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java Outdated Show resolved Hide resolved

danny0405 reviewed Oct 28, 2023

View reviewed changes

...rc/main/java/org/apache/hudi/common/table/read/HoodiePositionBasedFileGroupRecordBuffer.java Show resolved Hide resolved

danny0405 reviewed Oct 28, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java Show resolved Hide resolved

danny0405 reviewed Oct 28, 2023

View reviewed changes

hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/merge/SparkRecordMergingUtils.java Show resolved Hide resolved

Address comments and add docs

69bed1e

danny0405 reviewed Oct 30, 2023

View reviewed changes

hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java Outdated Show resolved Hide resolved

Add examples to demonstrate partial merging

5a7c85e

danny0405 approved these changes Oct 30, 2023

View reviewed changes

yihua added 3 commits October 30, 2023 14:57

Fix SparkFileFormatInternalRowReaderContext to use HoodieSparkParquet…

3651b2c

…Reader to read parquet log blocks

Fix style issues and clean up code

99af35c

Fix spark2 build

17a9b8d

apache deleted a comment from hudi-bot Oct 30, 2023

Keep vectorized reader disabled in test

45f06f3

yihua merged commit 26ecb28 into apache:master Oct 31, 2023

yihua mentioned this pull request Oct 8, 2024

Flink Hudi Support Partial Column Update #6598

Closed

4 tasks

Conversation

yihua commented Oct 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level

Documentation Update

Contributor's checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny0405 Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

yihua Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danny0405 Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

yihua Oct 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny0405 Oct 28, 2023

Choose a reason for hiding this comment

Uh oh!

yihua Oct 28, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danny0405 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yihua commented Oct 19, 2023 •

edited

Loading

yihua Oct 26, 2023 •

edited

Loading