Source deleted column with index, target did not delete column

[Jayjlchan]

What did you do?

source（mysql 5.7）deleted column with index, target(tidb 6.5.0) did not delete column, but DM did not report error message.
For example:

-- mysql1，mysql2(source1、source2)  and  tidb(target) is the same metadata

mysql> show create table tab2;
 CREATE TABLE `tab2` (
  `id` int(11) NOT NULL,
  `first_apply_proof_time` datetime DEFAULT NULL COMMENT '首次申请时间',
  `mn1` int(11) NOT NULL DEFAULT '10',
  `mn2` int(11) NOT NULL DEFAULT '10',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  KEY `idx1` (`mn1`),
  KEY `idx2` (`mn2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin 

-- mysql1，mysql2(source1、source2) 
mysql> ALTER TABLE sbtest.tab2 DROP COLUMN mn1,DROP COLUMN mn2;
Query OK, 0 rows affected (0.04 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> show create table tab2;
CREATE TABLE `tab2` (
  `id` int(11) NOT NULL,
  `first_apply_proof_time` datetime DEFAULT NULL COMMENT '首次申请时间',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

-- tidb(target)
mysql> show create table tab2;
 CREATE TABLE `tab2` (
  `id` int(11) NOT NULL,
  `first_apply_proof_time` datetime DEFAULT NULL COMMENT '首次申请时间',
  `mn1` int(11) NOT NULL DEFAULT '10',
  `mn2` int(11) NOT NULL DEFAULT '10',
  PRIMARY KEY (`id`) /*T![clustered_index] CLUSTERED */,
  KEY `idx1` (`mn1`),
  KEY `idx2` (`mn2`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin 

What did you expect to see?

The target column is deleted

What did you see instead?

The target column is not deleted, and DM did not report error message.

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

6.5.0

Upstream MySQL/MariaDB server version:

MySQL 5.7.19

Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

6.5.0

How did you deploy DM: tiup or manually?

tiup

Other interesting information (system version, hardware config, etc):

>
>

current status of DM cluster (execute query-status <task-name> in dmctl)

» query-status test650
{
    "result": true,
    "msg": "",
    "sources": [
        {
            "result": true,
            "msg": "",
            "sourceStatus": {
                "source": "mysql-replica-3307",
                "worker": "dm-172.16.201.206-8264",
                "result": null,
                "relayStatus": null
            },
            "subTaskStatus": [
                {
                    "name": "test650",
                    "stage": "Running",
                    "unit": "Sync",
                    "result": null,
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "4",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(bin.000001, 127901)",
                        "masterBinlogGtid": "8bf2dbc3-b1c1-11ed-ae6a-fa163e3df2cf:1-553",
                        "syncerBinlog": "(bin.000001, 127698)",
                        "syncerBinlogGtid": "8bf2dbc3-b1c1-11ed-ae6a-fa163e3df2cf:1-552",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false,
                        "binlogType": "remote",
                        "secondsBehindMaster": "0",
                        "blockDDLOwner": "",
                        "conflictMsg": "",
                        "totalRows": "4",
                        "totalRps": "0",
                        "recentRps": "0"
                    },
                    "validation": null
                }
            ]
        },
        {
            "result": true,
            "msg": "",
            "sourceStatus": {
                "source": "mysql-replica-3308",
                "worker": "dm-172.16.201.186-8264",
                "result": null,
                "relayStatus": null
            },
            "subTaskStatus": [
                {
                    "name": "test650",
                    "stage": "Running",
                    "unit": "Sync",
                    "result": null,
                    "unresolvedDDLLockID": "",
                    "sync": {
                        "totalEvents": "4",
                        "totalTps": "0",
                        "recentTps": "0",
                        "masterBinlog": "(bin.000001, 105325)",
                        "masterBinlogGtid": "2bb857c6-b1c3-11ed-983c-fa163e3df2cf:1-461",
                        "syncerBinlog": "(bin.000001, 105122)",
                        "syncerBinlogGtid": "2bb857c6-b1c3-11ed-983c-fa163e3df2cf:1-460",
                        "blockingDDLs": [
                        ],
                        "unresolvedGroups": [
                        ],
                        "synced": false,
                        "binlogType": "remote",
                        "secondsBehindMaster": "0",
                        "blockDDLOwner": "",
                        "conflictMsg": "",
                        "totalRows": "4",
                        "totalRps": "0",
                        "recentRps": "0"
                    },
                    "validation": null
                }
            ]
        }
    ]
}

[lance6716]

Please answer this question

[Jayjlchan]

Please answer this question

done

[lance6716]

it's a bit strange, can you provide the log? Also you can use https://asktug.com/tag/dm to get support from community.

[fubinzh]

Be able to reproduce the issue with v6.5.0. Is it bacause DDL is the last event (#8193)? I Delete colomn event is synced to downstream after inserting some data to the upstreams in my testing.

dm-worker-1.log
dm-worker-0.log

my task configuration:

name: test
task-mode: all
is-sharding: true
shard-mode: "optimistic"

target-database:
    host: "downstream-tidb"
    port: 4000
    user: "root"
    password: ""

mysql-instances:
    - source-id: "source-0"
      block-allow-list: "instance"
    - source-id: "source-1"
      block-allow-list: "instance"

block-allow-list:
    instance:
        do-dbs: ["test"]

[fubinzh]

/severity moderate