Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hotfix][cdc-connector][mysql] skip SchemaChangeEvents that were not included in capturedTableFilter. #2986

Merged
merged 1 commit into from
Jan 15, 2024

Conversation

lvyanquan
Copy link
Contributor

This closes #2981.

@lvyanquan
Copy link
Contributor Author

@ruanhang1993 @Jiabao-Sun PTAL.

@lvyanquan
Copy link
Contributor Author

lvyanquan commented Jan 10, 2024

Met some failures in CI, will try to fix it.
Fixed.

@lvyanquan
Copy link
Contributor Author

@leonardBang CC.

Copy link
Contributor

@leonardBang leonardBang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @lvyanquan for the fix, LGTM

@leonardBang leonardBang added this to the V3.1.0 milestone Jan 15, 2024
@leonardBang leonardBang merged commit 9abf2cf into apache:master Jan 15, 2024
17 checks passed
joyCurry30 pushed a commit to joyCurry30/flink-cdc-connectors that referenced this pull request Mar 22, 2024
@steveGuRen
Copy link

@joyCurry30 @leonardBang Logically, it is considered a bug when I have to read all tables in the schema, and the first execution of "alter table sql" is not triggered. However, since TableDiscoveryUtils loads all tables, it works. But in a SaaS environment, with so many tables(five hundred thousand, a tenant has their own tables), this will cause issues. Because we can't load all tables when have to discover the new Table with query sql by "show create sql " for every table.
So I fix the TableDiscoveryUtils so that just load a table in start or ddl change, and shouEmit logic updates too

@steveGuRen
Copy link

In the design of TiDB, I find that they take into account the scenario of having a large number of tables in a SaaS environment. Therefore, special attention is given to the performance of DDL operations in such contexts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Failed when a table that was not included in the source changed its schema.
3 participants