Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature][connector-v2][clickhouse] Support write cdc changelog event in clickhouse sink #3653

Merged
merged 4 commits into from
Dec 13, 2022
Merged

[feature][connector-v2][clickhouse] Support write cdc changelog event in clickhouse sink #3653

merged 4 commits into from
Dec 13, 2022

Conversation

hailin0
Copy link
Member

@hailin0 hailin0 commented Dec 5, 2022 •
edited
Loading

Purpose of this pull request

Close #3652

MergeTree Table Engine:

  • Support enable allowExperimentalLightweightDelete setting

  • CDC Events:

    • INSERT: INSERT INTO SQL
    • UPDATE_BEFORE: DELETE SQL
    • UPDATE_AFTER: INSERT INTO SQL
    • DELETE: DELETE SQL

  • ReplacingMergeTree Engine CDC Events:

    • INSERT: INSERT INTO SQL
    • UPDATE_BEFORE: ignore
    • UPDATE_AFTER: INSERT INTO SQL
    • DELETE: DELETE SQL

Other Table Engine:

  • CDC Events:
    • INSERT: INSERT INTO SQL
    • UPDATE_BEFORE: ignore
    • UPDATE_AFTER: ALTER TABLE UPDATE SQL
    • DELETE: ALTER TABLE DELETE SQL

Check list

@hailin0 hailin0 added cdc feature New feature labels Dec 5, 2022
@hailin0 hailin0 force-pushed the support-cdc-sink-for-clickhouse branch 2 times, most recently from d80d6ca to 5ef2d39 Compare December 7, 2022 02:06
@EricJoy2048 EricJoy2048 added this to the 2.3.0 milestone Dec 8, 2022
@hailin0 hailin0 force-pushed the support-cdc-sink-for-clickhouse branch from 5ef2d39 to bf39e88 Compare December 8, 2022 10:52
@hailin0 hailin0 marked this pull request as ready for review December 8, 2022 11:05
@hailin0
Copy link
Member Author

hailin0 commented Dec 8, 2022

@EricJoy2048 @Hisoka-X PTAL

Hisoka-X
Hisoka-X reviewed Dec 8, 2022
View reviewed changes
...onnectors/seatunnel/clickhouse/sink/client/executor/BufferReducedBatchStatementExecutor.java Outdated Show resolved Hide resolved
...nnectors/seatunnel/clickhouse/sink/client/executor/InsertOrUpdateBatchStatementExecutor.java Outdated
valueRowConverter.toExternal(record, updateStatement);
updateStatement.addBatch();
} else {
if (preChangeFlag != null && preChangeFlag) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the previous record is an update and this record is an insert, it seems unnecessary to execute the update statement.

@hailin0 hailin0 force-pushed the support-cdc-sink-for-clickhouse branch from bf39e88 to d546060 Compare December 8, 2022 15:33
@hailin0
[feature][connector-v2][clickhouse] Support write cdc changelog event…
76c2b86 
… in clickhouse sink

MergeTree Table Engine:
- Support enable `allowExperimentalLightweightDelete` setting
- CDC Events:
  - INSERT: INSERT INTO SQL
  - UPDATE_BEFORE: DELETE SQL
  - UPDATE_AFTER: INSERT INTO SQL
  - DELETE: DELETE SQL

- ReplacingMergeTree Engine CDC Events:
  - INSERT: INSERT INTO SQL
  - UPDATE_BEFORE: ignore
  - UPDATE_AFTER: INSERT INTO SQL
  - DELETE: DELETE SQL

Other Table Engine:
- CDC Events:
  - INSERT: INSERT INTO SQL
  - UPDATE_BEFORE: ignore
  - UPDATE_AFTER: ALTER TABLE UPDATE SQL
  - DELETE: ALTER TABLE DELETE SQL
@hailin0
update
727dc05
@hailin0 hailin0 force-pushed the support-cdc-sink-for-clickhouse branch from d546060 to 727dc05 Compare December 9, 2022 02:08
@hailin0 hailin0 requested a review from Hisoka-X December 9, 2022 06:29
@hailin0 hailin0 force-pushed the support-cdc-sink-for-clickhouse branch from 30bfeaa to a17e93f Compare December 9, 2022 06:53
Hisoka-X
Hisoka-X approved these changes Dec 13, 2022
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Hisoka-X Hisoka-X merged commit 6093c21 into apache:dev Dec 13, 2022
@hailin0 hailin0 mentioned this pull request Dec 15, 2022
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@EricJoy2048 EricJoy2048 EricJoy2048 approved these changes

@Hisoka-X Hisoka-X Hisoka-X approved these changes

Assignees
No one assigned
Labels
approved cdc feature New feature reviewed
Projects
None yet
Milestone
2.3.0
Development

Successfully merging this pull request may close these issues.

[Feature][CDC] Support RowKind(DELETE/UPDATE_BEFORE/UPDATE_AFTER) data row in Clickhouse sink
3 participants
@hailin0 @EricJoy2048 @Hisoka-X