Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schema_only mode seems to be slow when replicating database with a lot of tables. #507

Closed
subkanthi opened this issue Mar 20, 2024 · 1 comment · Fixed by #509
Closed
Labels
lightweight Issues related to Lightweight version qa-verified label to mark issues that were verified by QA
Milestone

Comments

@subkanthi
Copy link
Collaborator

There is a debezium configuration parameter schema.history.internal.store.only.captured.tables.ddl which should only include the ddl for the tables listed in the parameter.
But during testing it looks like a lot of time is spent during startup to retrieve the schema for all the tables.

Implementation:
Try to create a database with 15k tables with the configuration variables set in debezium.

@subkanthi subkanthi linked a pull request Mar 22, 2024 that will close this issue
@subkanthi
Copy link
Collaborator Author

subkanthi commented Mar 25, 2024

The following configuration works in limiting debezium to only snapshot the tables listed in the table.include.list

table.include.list: "sbtest.table1,sbtest.table2,sbtest.table3"
schema.history.internal.store.only.captured.tables.ddl: "true"
schema.history.internal.store.only.captured.databases.ddl: "true"

Regex:

"table.include.list","employees.table[1-3]$"

[debezium-mysqlconnector-embeddedconnector-change-event-source-coordinator] INFO io.debezium.relational.RelationalSnapshotChangeEventSource - Snapshot step 3 - Locking captured tables [employees.table1, employees.table2, employees.table3]

Before configuration with 15k tables:

[output_before.log](https://github.com/Altinity/clickhouse-sink-connector/files/14749835/output_before.log)

After configuration: limiting to 3 tables:
output.log

[debezium-mysqlconnector-embeddedconnector-change-event-source-coordinator] INFO io.debezium.relational.RelationalSnapshotChangeEventSource - Snapshot step 3 - Locking captured tables [sbtest.table1, sbtest.table2, sbtest.table3]

@svb-alt svb-alt added this to the 2.2.0 milestone Apr 4, 2024
@svb-alt svb-alt added lightweight Issues related to Lightweight version and removed 2.0.2 labels Apr 4, 2024
@svb-alt svb-alt modified the milestones: 2.2.0, 2.0.2 Apr 4, 2024
@Selfeer Selfeer added the qa-verified label to mark issues that were verified by QA label Apr 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lightweight Issues related to Lightweight version qa-verified label to mark issues that were verified by QA
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants