Skip to content

[Subtask]: flink-cdc partition table data inconsistency #23970

@jiangxinmeng1

Description

@jiangxinmeng1

Parent Issue

#23877

Detail of Subtask

测试步骤:

删除旧库(如果存在)

MYSQL_PWD="MOmo@202301" mysql --protocol=tcp -h 10.222.1.128 -P 3306 -u dump -e "DROP DATABASE IF EXISTS cdc_comprehensive_test;"

执行 DDL 建表(25张表,覆盖数值/字符串/日期/JSON/索引/约束/分区/多引擎/多字符集/ROW_FORMAT等)

cat scripts/types-test/mysql_comprehensive_ddl.sql | sed 's/DB/cdc_comprehensive_test/g' | MYSQL_PWD="MOmo@202301" mysql --protocol=tcp -h 10.222.1.128 -P 3306 -u dump

MYSQL_PWD="111" mysql --protocol=tcp -h 10.222.1.129 -P 6001 -u dump -e "DROP DATABASE IF EXISTS cdc_comprehensive_test;"

cat scripts/types-test/mo_comprehensive_ddl.sql | sed 's/DB/cdc_comprehensive_test/g' | MYSQL_PWD="111" mysql --protocol=tcp -h 10.222.1.129 -P 6001 -u dump

MySQL 插入测试数据

每张表100行,含边界值(MIN/MAX/NULL/空串/emoji/多语言/精度极限等)+ 随机数据

python3 scripts/types-test/insert_comprehensive_data.py --db cdc_comprehensive_test --rows 100 --seed 42 | MYSQL_PWD="MOmo@202301" mysql --protocol=tcp -h 10.222.1.128 -P 3306 -u dump

启动 Producer(MySQL binlog → Kafka)

./scripts/producer-realtime.sh --db cdc_comprehensive_test --tables all --src-host 10.222.1.128 --source-user du --source-password "MOmo@202301" --topic t10

启动 Consumer

./scripts/consumer.sh --db cdc_comprehensive_test --dest-host 10.222.1.129 --consumer-batch-si 2000 --topic t10 --group group-10

数据量验证:# 方法一:用验证脚本(含行数/聚合/采样逐列对比)
./scripts/types-test/validate_comprehensive_sync.sh
--mysql-db cdc_comprehensive_test --mo-db cdc_comprehensive_test
--mysql-host 10.222.1.128 --mysql-port 3306 --mysql-user dump --mysql-password 'MOmo@202301'
--mo-host 10.222.1.129 --mo-port 6001 --mo-user dump --mo-password '111'

Describe implementation you've considered

No response

Additional information

No response

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions