[INLONG-4814][Sort][Manager] Declare the primary key for sink table when the query contains update/delete#4882
Merged
EMsnap merged 6 commits intoapache:masterfrom Aug 2, 2022
Conversation
Contributor
|
@Greedyu
|
Contributor
|
@yunqingmoswu @Greedyu
|
Contributor
Author
I thought this was just a bug before, but I didn't expect this ability to be purposely not supported. |
healchow
reviewed
Jul 10, 2022
...t-common/src/test/java/org/apache/inlong/sort/protocol/node/load/ClickHouseLoadNodeTest.java
Outdated
Show resolved
Hide resolved
EMsnap
reviewed
Jul 20, 2022
Member
EMsnap
left a comment
There was a problem hiding this comment.
please add some commentary on the cost of clickhouse update when added primary key
healchow
approved these changes
Jul 20, 2022
a6407c1 to
2a4a11e
Compare
haibo-duan
reviewed
Jul 31, 2022
...ng-sort/sort-end-to-end-tests/src/test/java/org/apache/inlong/sort/tests/ClickHouseCase.java
Show resolved
Hide resolved
thesumery
reviewed
Aug 1, 2022
.../sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/ClickHouseLoadNode.java
Outdated
Show resolved
Hide resolved
thesumery
reviewed
Aug 1, 2022
.../sort-common/src/main/java/org/apache/inlong/sort/protocol/node/load/ClickHouseLoadNode.java
Show resolved
Hide resolved
…uery contains update/delete record.
2. Add Flink cdc from mysql to clickHouse test. including update() and delete() methods
69bf9aa to
66cbc06
Compare
Contributor
Author
Add in ClickHouseLoadNode class |
gong
reviewed
Aug 1, 2022
...-sort/sort-connectors/jdbc/src/main/java/org/apache/inlong/sort/jdbc/table/JdbcDialects.java
Outdated
Show resolved
Hide resolved
gong
reviewed
Aug 1, 2022
inlong-sort/sort-end-to-end-tests/src/test/resources/flinkSql/clickhouse_test.sql
Outdated
Show resolved
Hide resolved
2. JdbcDialects removes clickhouse dialect, specified by table creation
Contributor
|
LGTM |
bruceneenhl
pushed a commit
to bruceneenhl/inlong
that referenced
this pull request
Aug 12, 2022
…hen the query contains update/delete (apache#4882)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Handling exceptions: please declare primary key for sink table when query contains update/delete record.
Briefly describe ideas
The problem is that with flink JDBC SQL you have to specify the primary key.

When the Sort module sees the processing of primaryKey, it should be a problem with the parameters in the node. Gotta go check it out

Debug test locates that the primaryKey in the file read by sort is indeed empty
To query the source of assignment of GroupInfo.StreamInfo.nodes, there are only CreateSortConfigListenerV2 and CreateStreamSortConfigListener classes, and they are both created by the createNodesForStream() method.

Test after bug fixes