Skip to content

[INLONG-6842][Sort] Improve mysql-cdc2.0 to support tables without primary key#6859

Merged
dockerzhang merged 3 commits intoapache:masterfrom
EMsnap:6842
Dec 14, 2022
Merged

[INLONG-6842][Sort] Improve mysql-cdc2.0 to support tables without primary key#6859
dockerzhang merged 3 commits intoapache:masterfrom
EMsnap:6842

Conversation

@EMsnap
Copy link
Member

@EMsnap EMsnap commented Dec 13, 2022

Prepare a Pull Request

Motivation

Fixes #6842

Modifications

A minor modification and improvemnt for Dblog : https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b

1、remove the constraint in mysql-cdc 2.0 for primary key
2、for those tables which don't contain primary key, split the whole table as a chunk during split stage
3、since there's no need for the MySqlSourceEnumerator to upsert binlog with the snapshot data for a whole table
split , just let the highwatermark equals to the lowwatermark to escape the upsert stage.

Verifying this change

run AllmigrateTest

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

@EMsnap EMsnap changed the title [INLONG-6842][Sort] Improve mysql-cdc2.0 to support faster snapshot read for tables without primary key [INLONG-6842][Sort] Improve mysql-cdc2.0 to support tables without primary key Dec 13, 2022
@gong
Copy link
Contributor

gong commented Dec 13, 2022

manager need to modify. Because manager set scan.incremental.snapshot.enabled param.

@EMsnap
Copy link
Member Author

EMsnap commented Dec 13, 2022

manager need to modify. Because manager set scan.incremental.snapshot.enabled param.

will submit another pr for manager modification

@dockerzhang dockerzhang merged commit cc6fcb6 into apache:master Dec 14, 2022
featzhang pushed a commit to featzhang/inlong that referenced this pull request Dec 17, 2022
…imary key (apache#6859)

Co-authored-by: stingpeng <stingpeng@tencent.com>
@EMsnap EMsnap deleted the 6842 branch May 30, 2023 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improve] improve mysql-cdc2.0 to support faster snapshot read

5 participants