feat(connector): support hudi sink #8824

wenym1 · 2023-03-28T09:22:52Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

In this PR, we support a very trivial hudi sink. The hudi sink only supports hudi MOR table, has only one parallelism, and is sinking to only one file group, with file id named risingwave-file-id. For this file group, the initial write is regarded as insert and write the initial base parquet file, and all subsequent updates are regarded as update, and write the log file.

An example SQL to create the hudi sink is

create sink hudi_sink from s1
with (
    connector = 'hudi',
    type='upsert',
    primary_key = 'seq_id',
    base.path = 's3a://my-bucket',
    table.name = 'hudi_tbl1',
    s3.endpoint = 'https://s3.ap-southeast-1.amazonaws.com/',
    s3.access.key = 'xxxxxxxxx',
    s3.secret.key = 'xxxxxxxx',
);

Checklist For Contributors

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
I have demonstrated that backward compatibility is not broken by breaking changes and created issues to track deprecated features to be removed in the future. (Please refer to the issue)
All checks passed in ./risedev check (or alias, ./risedev c)

Checklist For Reviewers

I have requested macro/micro-benchmarks as this PR can affect performance substantially, and the results are shown.

Documentation

My PR DOES NOT contain user-facing changes.

Click here for Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note

github-actions

license-eye has totally checked 3147 files.

Valid	Invalid	Ignored	Fixed
1460	8	1679	0

Click to see the invalid file list

java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HoodieRisingWaveMergeOnReadTable.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HoodieRisingWaveWriter.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSink.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkFactory.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveCreateHandleFactory.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveInsertCommitActionExecutor.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveUpsertPreppedDeltaCommitActionExecutor.java
java/connector-node/risingwave-sink-hudi/src/test/java/com/risingwave/connector/HudiSinkFactoryTest.java

...gwave-sink-hudi/src/main/java/com/risingwave/connector/HoodieRisingWaveMergeOnReadTable.java

...node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HoodieRisingWaveWriter.java

java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSink.java

...nector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkFactory.java

...singwave-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveCreateHandleFactory.java

...e-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveInsertCommitActionExecutor.java

...src/main/java/com/risingwave/connector/RisingWaveUpsertPreppedDeltaCommitActionExecutor.java

...or-node/risingwave-sink-hudi/src/test/java/com/risingwave/connector/HudiSinkFactoryTest.java

github-actions

license-eye has totally checked 3246 files.

Valid	Invalid	Ignored	Fixed
1497	9	1740	0

Click to see the invalid file list

java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HoodieRisingWaveMergeOnReadTable.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HoodieRisingWaveWriter.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSink.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkConfig.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkFactory.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveCreateHandleFactory.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveInsertCommitActionExecutor.java
java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/RisingWaveUpsertPreppedDeltaCommitActionExecutor.java
java/connector-node/risingwave-sink-hudi/src/test/java/com/risingwave/connector/HudiSinkFactoryTest.java

...nnector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkConfig.java

… field

codecov · 2023-05-09T01:37:15Z

Codecov Report

Merging #8824 (f28ed87) into main (33b583e) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #8824      +/-   ##
==========================================
- Coverage   70.98%   70.97%   -0.01%     
==========================================
  Files        1241     1241              
  Lines      207549   207549              
==========================================
- Hits       147329   147314      -15     
- Misses      60220    60235      +15

Flag	Coverage Δ
rust	`70.97% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/connector/src/sink/remote.rs	`58.12% <ø> (ø)`
...rc/frontend/src/optimizer/plan_node/stream_sink.rs	`70.70% <100.00%> (ø)`

... and 7 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

liurenjie1024

A simple implementation, generally LGTM, but maybe we should add some test case?

liurenjie1024 · 2023-05-25T09:17:02Z

...nector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkFactory.java

+ * limitations under the License.
+ */
+
+package com.risingwave.connector;


Suggested change

package com.risingwave.connector;

package com.risingwave.connector.hudi;

liurenjie1024 · 2023-05-25T09:28:57Z

...nnector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkRowMap.java

+            if (delete != null) {
+                map.put(key, HudiSinkRowOp.deleteOp(key));
+            } else {
+                map.remove(key);


In what case this may be null?

liurenjie1024 · 2023-05-25T09:33:16Z

java/connector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSink.java

+                                KeyGenUtils.getRecordKey(rec, this.recordKeyField, false), "");
+                switch (row.getOp()) {
+                    case INSERT:
+                        sinkRowMap.insert(


In such a simple case, I think maybe we don't need this row map. Hudi's log file can be used to append change logs. This way we can append records when writing, rather than writing all in clear.

tabVersion · 2023-12-26T08:14:55Z

src/frontend/src/optimizer/plan_node/stream_sink.rs

+                    Some(s) if s == "iceberg" || s == "hudi" => {
                        // iceberg with multiple parallelism will fail easily with concurrent commit
                        // on metadata
                        // TODO: reset iceberg sink to have multiple parallelism


[Re: lines 78 to 82]

please change the info! contents also

See this comment inline on Graphite.

feat(connector): support hudi sink

0745c98

github-actions bot added the type/feature label Mar 28, 2023

github-actions bot reviewed Mar 28, 2023

View reviewed changes

William Wen and others added 6 commits April 10, 2023 21:51

Merge branch 'main' into yiming/support-hudi-sink

6866f6a

refactor(sink): unify s3 config for iceberg and delta lake sink

8c6a7c3

Merge branch 'yiming/unify-s3-config' into yiming/support-hudi-sink

8549150

not set s3-common as provided globally

40307e8

Merge branch 'yiming/unify-s3-config' into yiming/support-hudi-sink

b75cb1c

support write to s3

d4dda05

github-actions bot reviewed Apr 12, 2023

View reviewed changes

...nnector-node/risingwave-sink-hudi/src/main/java/com/risingwave/connector/HudiSinkConfig.java Show resolved Hide resolved

wenym1 and others added 19 commits April 12, 2023 03:35

add hadoop-client dependency

65a0c7f

Merge branch 'yiming/unify-s3-config' into yiming/support-hudi-sink

3a8c1a9

add license header

51f5831

Merge branch 'main' into yiming/unify-s3-config

a98b55d

test(connector-node): set the pyspark version as 3.3 in integration test

40d4d0f

Merge branch 'yiming/set-pyspark-version' into yiming/unify-s3-config

ab88ee8

Merge branch 'yiming/unify-s3-config' into yiming/support-hudi-sink

022d5a7

add validate hudi table

8862a31

upgrade avro version to 1.9.1

fcc47a7

Merge branch 'main' into yiming/set-pyspark-version

2e0a4e0

Merge branch 'yiming/set-pyspark-version' into yiming/unify-s3-config

89d3dbb

Merge branch 'main' into yiming/unify-s3-config

ef2450a

recovery delta lake config constructor to set non-null requirement on…

5a66d4f

… field

disable split in the first insert

f0e87cd

fix unused import

278c419

Merge branch 'main' into yiming/support-hudi-sink

770a8eb

Merge commit 'e16690c' into yiming/unify-s3-config

2425502

Merge branch 'main' into yiming/unify-s3-config

62e0d77

Merge branch 'yiming/unify-s3-config' into yiming/support-hudi-sink

f28ed87

wenym1 marked this pull request as ready for review May 9, 2023 01:10

wenym1 requested review from liurenjie1024, hzxa21, tabVersion and xx01cyx May 9, 2023 01:11

liurenjie1024 reviewed May 25, 2023

View reviewed changes

tabVersion reviewed Dec 26, 2023

View reviewed changes

wenym1 marked this pull request as draft May 16, 2024 06:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(connector): support hudi sink #8824

feat(connector): support hudi sink #8824

wenym1 commented Mar 28, 2023 •

edited

github-actions bot left a comment

github-actions bot left a comment

codecov bot commented May 9, 2023 •

edited

liurenjie1024 left a comment

liurenjie1024 May 25, 2023

liurenjie1024 May 25, 2023

liurenjie1024 May 25, 2023

tabVersion Dec 26, 2023

	package com.risingwave.connector;
	package com.risingwave.connector.hudi;

feat(connector): support hudi sink #8824

Are you sure you want to change the base?

feat(connector): support hudi sink #8824

Conversation

wenym1 commented Mar 28, 2023 • edited

What's changed and what's your intention?

Checklist For Contributors

Checklist For Reviewers

Documentation

Types of user-facing changes

Release note

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

codecov bot commented May 9, 2023 • edited

Codecov Report

liurenjie1024 left a comment

Choose a reason for hiding this comment

liurenjie1024 May 25, 2023

Choose a reason for hiding this comment

liurenjie1024 May 25, 2023

Choose a reason for hiding this comment

liurenjie1024 May 25, 2023

Choose a reason for hiding this comment

tabVersion Dec 26, 2023

Choose a reason for hiding this comment

wenym1 commented Mar 28, 2023 •

edited

codecov bot commented May 9, 2023 •

edited