What did you do?
We hit a failure in storage integration test case ddl_for_split_tables_with_random_merge_and_split.
Key timeline from logs:
- Storage consumer executed rename DDL (
table_3 -> table_103) successfully:
logs/cdc_storage_consumer.log:7187
commitTs=464655649939914804
- query:
RENAME TABLE test.table_3 TO test.table_103
- After that, consumer still processed older DMLs for old table name
test.table_3:
logs/cdc_storage_consumer.log:7223
logs/cdc_storage_consumer.log:7225
- observed older commitTs, e.g.
464655649677770765, 464655649716830245
- Generated SQL still targeted old table name, then failed:
logs/cdc_storage_consumer.log:7423
logs/cdc_storage_consumer.log:7424
logs/cdc_storage_consumer.log:7428
logs/cdc_storage_consumer.log:7437
Observed MySQL error:
Error 1146 (42S02): Table 'test.table_3' doesn't exist
sync_diff failed afterwards as a downstream consequence:
logs/sync_diff/output/sync_diff.log:12
- error:
the source has no table to be compared. target-table is test.table_2
What did you expect to see?
- For the same table, DDL and DML application should remain commitTs-consistent.
- Older DML for old table name should not be executed after rename DDL has taken effect.
- Storage consumer should keep running and integration should pass
sync_diff.
What did you see instead?
- Rename DDL was applied first.
- Older DML for old table name was still executed.
- Consumer hit MySQL 1146 and exited.
sync_diff failed due to downstream divergence after consumer exit.
What did you do?
We hit a failure in storage integration test case
ddl_for_split_tables_with_random_merge_and_split.https://prow.tidb.net/jenkins/blue/organizations/jenkins/pingcap%2Fticdc%2Fpull_cdc_storage_integration_heavy/detail/pull_cdc_storage_integration_heavy/525/pipeline/449/
logs/bundle.Key timeline from logs:
table_3 -> table_103) successfully:logs/cdc_storage_consumer.log:7187commitTs=464655649939914804RENAME TABLE test.table_3 TO test.table_103test.table_3:logs/cdc_storage_consumer.log:7223logs/cdc_storage_consumer.log:7225464655649677770765,464655649716830245logs/cdc_storage_consumer.log:7423logs/cdc_storage_consumer.log:7424logs/cdc_storage_consumer.log:7428logs/cdc_storage_consumer.log:7437Observed MySQL error:
Error 1146 (42S02): Table 'test.table_3' doesn't existsync_difffailed afterwards as a downstream consequence:logs/sync_diff/output/sync_diff.log:12the source has no table to be compared. target-table is test.table_2What did you expect to see?
sync_diff.What did you see instead?
sync_difffailed due to downstream divergence after consumer exit.