Remove table and clean up redundant state info when scanning newly added table#1134
Conversation
65ac40c to
cce5830
Compare
|
Hi @qidian99, I wonder why you PR didn't merged into master? |
|
@leonardBang Do you have time to review this PR? |
|
Hi, @qidian99 . Could you rebase the master branch? Thanks~ |
cce5830 to
d76935b
Compare
leonardBang
left a comment
There was a problem hiding this comment.
Thanks @qidian99 and @ruanhang1993 for the contribution, I left some coments
...ain/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java
Outdated
Show resolved
Hide resolved
...ain/java/com/ververica/cdc/connectors/mysql/source/assigners/MySqlSnapshotSplitAssigner.java
Show resolved
Hide resolved
leonardBang
left a comment
There was a problem hiding this comment.
Thanks @ruanhang1993 for the update, LGTM
…rting from checkpoint
ec44af1 to
3a62bd7
Compare
|
1、I configured MySQL tables A and B to synchronize to a pipeline in Starrocks using the configuration file yaml Task. |
As mentioned in Issue#913, when a task is stopped and its configuration changed, there might be redundant table information remaining in the states of SplitAssigner and SourceReader.
The current functionality of scanning newly added tables will removing and cleaning up table info in the previous savepoint according to the new configuration.
The changes proposed in this PR is as follows:
In MySqlSplitAssigner, it will apply the new filter in the new configuration and remove redundant table info
In MySqlSourceReader, it will sieve out splits that do not match the new filter.