Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Connector-V2] Support iceberg sink #6265

Merged
merged 1 commit into from
Feb 19, 2024
Merged

[Feature][Connector-V2] Support iceberg sink #6265

merged 1 commit into from
Feb 19, 2024

Conversation

sunxiaojian
Copy link
Contributor

@sunxiaojian sunxiaojian commented Jan 22, 2024 •
edited

Related issues : #6198

Check list

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 2 times, most recently from 7338bc5 to b2569b4 Compare January 22, 2024 12:10
@sunxiaojian
Copy link
Contributor Author

Related PR : #5490 / #5072

@sunxiaojian
Copy link
Contributor Author

@hailin0 PTAL

@hailin0
Copy link
Contributor

hailin0 commented Jan 22, 2024

image

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 8 times, most recently from ca300b8 to 09a85ee Compare January 25, 2024 04:23
@sunxiaojian
Copy link
Contributor Author

image

@hailin0 done

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 5 times, most recently from d50148a to c4a7a22 Compare January 25, 2024 09:06
@sunxiaojian
Copy link
Contributor Author

@hailin0 Can you help me retry CI？

@sunxiaojian sunxiaojian force-pushed the iceberg-sink branch 2 times, most recently from 516b52c to 10be48f Compare January 27, 2024 14:15
@hailin0
Copy link
Contributor

hailin0 commented Jan 29, 2024

https://github.com/sunxiaojian/seatunnel/runs/20929811031

Requires you to trigger actions

@sunxiaojian

@sunxiaojian
Copy link
Contributor Author

https://github.com/sunxiaojian/seatunnel/runs/20929811031

Requires you to trigger actions

@sunxiaojian

@hailin0 done

@sunxiaojian
Copy link
Contributor Author

@hailin0 @ic4y PTAL

@sunxiaojian sunxiaojian mentioned this pull request Feb 6, 2024
Closed
3 tasks
hailin0
hailin0 reviewed Feb 18, 2024
View reviewed changes
...-connector-v2-e2e/connector-iceberg-e2e/src/test/resources/iceberg/mysql_cdc_to_iceberg.conf
}

source {
MySQL-CDC {
Copy link
Contributor

@hailin0 hailin0 Feb 18, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunxiaojian
Is it possible to use fake for testing cdc

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-clickhouse-e2e/src/test/resources/clickhouse_sink_cdc_changelog_case2.conf#L35
https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources/text/fake_to_local_file_with_multiple_table.conf

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sunxiaojian Is it possible to use fake for testing cdc

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-clickhouse-e2e/src/test/resources/clickhouse_sink_cdc_changelog_case2.conf#L35 https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources/text/fake_to_local_file_with_multiple_table.conf

@hailin0 The main purpose of using MySQL CDC is to test the upsert mode, which is closer to the actual scenario

@hailin0 hailin0 added feature New feature iceberg labels Feb 18, 2024
hailin0
hailin0 approved these changes Feb 19, 2024
View reviewed changes
Copy link
Contributor

@hailin0 hailin0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Hisoka-X
Hisoka-X approved these changes Feb 19, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Hisoka-X Hisoka-X merged commit 18d3e86 into apache:dev Feb 19, 2024
10 checks passed
@hailin0 hailin0 mentioned this pull request Feb 19, 2024
Closed
5 tasks
@hailin0 hailin0 linked an issue Mar 20, 2024 that may be closed by this pull request
[Feature][Connector-v2] Support iceberg sink #2645
Closed
3 tasks
@jtchen-study
Copy link

@sunxiaojian this kind of iceberg sink writing to upsert table will generated a lot of small data files? Has any solution to sovle it ?

4chicat pushed a commit to nexr/incubator-seatunnel that referenced this pull request Apr 9, 2024
@sunxiaojian @4chicat
[Feature] Supports iceberg sink apache#6198 (apache#6265)
3a616db
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hailin0 hailin0 hailin0 approved these changes

@Hisoka-X Hisoka-X Hisoka-X approved these changes

Assignees
No one assigned
Labels
approved feature New feature iceberg reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature][Connector-v2] Support iceberg sink
4 participants
@sunxiaojian @hailin0 @jtchen-study @Hisoka-X