[flink][bug] MergeIntoAction and DeleteAction work abnormally with partial update merge engine by yuzelin · Pull Request #829 · apache/paimon

yuzelin · 2023-04-04T09:55:31Z

Purpose

Currently, the partial update merge engine cannot accept -U/-D records, thus actions that may produce -U/-D records will produce error results. This PR try to solve it.

Tests

DeleteActionITCase
MergeIntoActionITCase

API and Format

No.

Documentation

Add hint to how-to#writing tables

…rtial update merge engine

JingsongLi · 2023-04-04T10:03:16Z

docs/content/how-to/writing-tables.md

 ```

+{{< hint info >}}
+If you have set `merge-engine` = `partial-update`, it will be disabled temporarily. 


aggregate merge-engine also?

JingsongLi · 2023-04-04T10:03:56Z

docs/content/how-to/writing-tables.md

 ```

+{{< hint info >}}
+If you have set `merge-engine` = `partial-update`, it will be disabled temporarily. 


I can not get it will be disabled temporarily

I want to explain that when the actions running, the merge engine settings are ignored; when the actions finishied, the merge engine settings still works.

But just dynamic options, no affect to file system.

JingsongLi · 2023-04-04T10:04:26Z

paimon-core/src/main/java/org/apache/paimon/table/AbstractFileStoreTable.java

    public FileStoreTable copy(Map<String, String> dynamicOptions) {
-        // check option is not immutable
        Map<String, String> options = new HashMap<>(tableSchema.options());
-        dynamicOptions.forEach(


don't remove this, create an another internal method.

You mean in this internal method, we can still check immutable options changing except merge engine?

I mean create a new method internalCopyWithoutCheck.

JingsongLi · 2023-04-06T02:05:27Z

paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/ActionBase.java

+     * want these records to be sunk directly. This method is a workaround. Actions that may produce
+     * -U/-D records can disable merge engine settings when running.
+     */
+    protected void resetMergeEngine() {


maybe not just reset merge engine, we need to force compaction too.

yuzelin · 2023-04-06T06:09:14Z

JingsongLi · 2023-04-07T02:30:05Z

paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/ActionBase.java

+    protected void forceSinking() {
+        Map<String, String> dynamicOptions = new HashMap<>();
+        dynamicOptions.put(
+                CoreOptions.MERGE_ENGINE.key(), CoreOptions.MergeEngine.DEDUPLICATE.toString());


only to non-DEDUPLICATE engines. Need to set FULL_COMPACTION_DELTA_COMMITS.

JingsongLi · 2023-04-07T02:31:11Z

paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/ActionBase.java

+     * want these records to be sunk directly. This method is a workaround. Actions that may produce
+     * -U/-D records can call this to disable merge engine settings and force compaction.
+     */
+    protected void forceSinking() {


changeIgnoreMergeEngine?

JingsongLi · 2023-04-07T02:32:05Z

docs/content/how-to/writing-tables.md

 ```

+{{< hint info >}}
+If you have set `merge-engine` = `partial-update` or `aggregation`, it will be reset by dynamic options 


remove this? I cannot understand it from user side.

Agree. I think it's not important for users to know.

JingsongLi · 2023-04-07T02:32:05Z

docs/content/how-to/writing-tables.md

 Important table properties setting:
 1. Only [primary key table]({{< ref "concepts/primary-key-table" >}}) supports this feature.
 2. The action won't produce UPDATE_BEFORE, so it's not recommended to set 'changelog-producer' = 'input'.
+3. If you have set `merge-engine` = `partial-update` or `aggregation`, it will be reset by dynamic options


remove this? I cannot understand it from user side.

JingsongLi

+1

sonarqubecloud · 2023-04-07T07:30:09Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication

[core] Renaming streaming-read-from to streaming-read-mode (apache#778) parent 4a56565 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194324 +0800 parent 4a56565 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194307 +0800 [core] Renaming streaming-read-from to streaming-read-mode(apache#778) [core] Renaming streaming-read-from to streaming-read-mode (apache#778) [core] generate doc (apache#778) [core] add test (apache#778) Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> [core] add test (apache#778) [core] commit (apache#778) [doc] Move Watermark and Bounded Stream to Append only table page [deploy] Publish paimon-shade to snapshot [core] Introduce SecurityContext (apache#834) [deps] Use paimon-shade dependencies (apache#838) [Document] Adds the Trino feature that create/drop table, create/drop database in the overview document (apache#835) [docs] Add document for MySqlSyncTableAction (apache#828) [doc] Document file format (apache#840) [ci] Add licensing check (apache#839) [common] Use caffeine to replace with guava cache (apache#665) [release] Added incubating name in release file (apache#845) [doc] add flink 1.17 version in Engines Overview (apache#847) [hive] Dropping table deletes table directory to avoid schema in filesystem exists (apache#772) [hive] Support hive-conf-dir and hadoop-conf-dir option in hive catalog (apache#831) [hotfix] Do not swallow InterruptedException in SnapshotManager [core][flink] Refactor BucketComputer and StreamPartitioner to gather bucketing / partitioning core logic in the same class (apache#844) [doc] Document Java API (apache#849) [README] Document mark generated-sources as Sources Root [flink][bug] MergeIntoAction and DeleteAction work abnormally with partial update merge engine (apache#829) [hotfix] Set directExecutor to caffeine Cache Resolve the problem which Sonar reported that InterruptedException need to re-throw or re-interrupt (apache#862) [hotfix] Enable StreamingReadWriteTableWithKafkaLogITCase.testReadInsertOnlyChangelogFromEnormousTimestamp (apache#854) [test] Migrate MinioTestContainer to paimon-s3 (apache#869) [flink] Assign splits evenly in StaticFileStoreSplitEnumerator (apache#856) [core] Compacted StartingScanner is incorrect (apache#857) [core] revert flink.md(apache#778) [core] change dose to does (apache#778)

[core] Add streaming read mode from option (apache#778) [core] Renaming streaming-read-from to streaming-read-mode (apache#778) parent 4a56565 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194324 +0800 parent 4a56565 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681139027 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681194307 +0800 [core] Renaming streaming-read-from to streaming-read-mode(apache#778) [core] Renaming streaming-read-from to streaming-read-mode (apache#778) [core] generate doc (apache#778) [core] add test (apache#778) Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> [core] add test (apache#778) [core] commit (apache#778) [doc] Move Watermark and Bounded Stream to Append only table page [deploy] Publish paimon-shade to snapshot [core] Introduce SecurityContext (apache#834) [deps] Use paimon-shade dependencies (apache#838) [Document] Adds the Trino feature that create/drop table, create/drop database in the overview document (apache#835) [docs] Add document for MySqlSyncTableAction (apache#828) [doc] Document file format (apache#840) [ci] Add licensing check (apache#839) [common] Use caffeine to replace with guava cache (apache#665) [release] Added incubating name in release file (apache#845) [doc] add flink 1.17 version in Engines Overview (apache#847) [hive] Dropping table deletes table directory to avoid schema in filesystem exists (apache#772) [hive] Support hive-conf-dir and hadoop-conf-dir option in hive catalog (apache#831) [hotfix] Do not swallow InterruptedException in SnapshotManager [core][flink] Refactor BucketComputer and StreamPartitioner to gather bucketing / partitioning core logic in the same class (apache#844) [doc] Document Java API (apache#849) [README] Document mark generated-sources as Sources Root [flink][bug] MergeIntoAction and DeleteAction work abnormally with partial update merge engine (apache#829) [hotfix] Set directExecutor to caffeine Cache Resolve the problem which Sonar reported that InterruptedException need to re-throw or re-interrupt (apache#862) [hotfix] Enable StreamingReadWriteTableWithKafkaLogITCase.testReadInsertOnlyChangelogFromEnormousTimestamp (apache#854) [test] Migrate MinioTestContainer to paimon-s3 (apache#869) [flink] Assign splits evenly in StaticFileStoreSplitEnumerator (apache#856) [core] Compacted StartingScanner is incorrect (apache#857) [core] revert flink.md(apache#778) [core] change dose to does (apache#778) parent 2a62022 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226783 +0800 parent 2a62022 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226730 +0800 parent 2a62022 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226598 +0800 [core] Add streaming read from option (apache#778) [core] add test (apache#778) Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-core/src/main/java/org/apache/paimon/CoreOptions.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> Update paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java Co-authored-by: Nicholas Jiang <programgeek@163.com> [core] add test (apache#778) merger sdda 这是一个 3 个提交的组合。 [core] commit1 (apache#778) [core] commit2 (apache#778) [core] commit3 (apache#778) [core] do commit 1(apache#778) [core] do commit 2(apache#778) sdda 这是一个 3 个提交的组合。 [core] commit1 (apache#778) [core] commit2 (apache#778) [core] commit3 (apache#778) [core] Add streaming read mode option (apache#778) parent 3acede2 author Alibaba-HZY <daowu.hzy@cainiao.com> 1681226824 +0800 committer Alibaba-HZY <daowu.hzy@cainiao.com> 1681226824 +0800 [core] Add streaming read from option (apache#778) [core] revert flink.md (apache#778) Delete AbstractChannelComputer.java delete

[flink][bug] MergeIntoAction and DeleteAction work abnormally with pa…

3c29f24

…rtial update merge engine

yuzelin force-pushed the action-partial-update branch from 2ab2441 to 3c29f24 Compare April 4, 2023 09:59

JingsongLi reviewed Apr 4, 2023

View reviewed changes

JingsongLi reviewed Apr 6, 2023

View reviewed changes

yuzelin force-pushed the action-partial-update branch from 60f093e to a230d37 Compare April 6, 2023 03:11

[fix] address review comments

1d1a204

yuzelin force-pushed the action-partial-update branch from d260585 to 1d1a204 Compare April 6, 2023 03:19

[fix] fix error test

e259a30

JingsongLi reviewed Apr 7, 2023

View reviewed changes

[fix] address review comments

5f4e9e0

yuzelin force-pushed the action-partial-update branch from 3cb0bd7 to 5f4e9e0 Compare April 7, 2023 02:51

JingsongLi approved these changes Apr 7, 2023

View reviewed changes

[fix] fix test

fa02a1b

yuzelin force-pushed the action-partial-update branch from 61f0161 to fa02a1b Compare April 7, 2023 07:28

JingsongLi merged commit fb9babb into apache:master Apr 7, 2023

yuzelin deleted the action-partial-update branch April 7, 2023 10:14

Conversation

yuzelin commented Apr 4, 2023

Purpose

Tests

API and Format

Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuzelin commented Apr 6, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants