after REORGANIZE PARTITION DDL , ticdc in error and lag increased

[AkiraXie]

What did you do?

1. open tidb changefeed
2. ALTER TABLE xxx REORGANIZE PARITITION xxxxx

What did you expect to see?

No response

What did you see instead?

1. [error="[CDC:ErrChangefeedUnretryable]changefeed is in unretryable state, please check the error message, and you should manually handle it: invalid connection"] [errorVerbose="[CDC:ErrChangefeedUnretryable]changefeed is in unretryable state, please check the error message, and you should manually handle it: invalid connection
   

2. after resume, changefeed checkpoint cannnot advance
   

3. after rolling restart ticdc , checkpoint cannnot advance yet , and ticdc in error due to DDL not reentrant

[2023/03/24 12:11:19.500 +08:00] [WARN] [mysql_ddl_sink.go:126] ["Execute DDL with error, retry later"] [startTs=440288719245410374] [ddl="ALTER TABLE `partition360`.`partition360_6` 

REORGANIZE PARTITION `p20210516`,`p20210517` INTO (PARTITION `p20210517` VALUES LESS THAN (_UTF8MB4'20210518'))"] [namespace=] 

[changefeed=] [error="Error 1567 (HY000): Incorrect partition name"]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

sh-5.1# /cdc version
Release Version: v7.0.0
Git Commit Hash: d5fd440
Git Branch: heads/refs/tags/v7.0.0
UTC Build Time: 2023-03-22 10:48:06
Go Version: go version go1.20.2 linux/amd64
Failpoint Build: false

[AkiraXie]

/severity major

[hicqu]

Owner keeps printing this:

[2023/03/23 22:54:06.845 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:06.896 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:06.946 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:06.996 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.046 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.095 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.145 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.196 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.245 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.296 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.346 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.396 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.446 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.495 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.545 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.595 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.645 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.695 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.745 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.796 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.845 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681] [2023/03/23 22:54:07.895 +08:00] [WARN] [replication_manager.go:548] ["schedulerv3: cannot advance checkpoint since missing span"] [namespace=default] [changefeed=cdc-test-01] [t
ableSpanFound=false] [tableSpanStartFound=false] [tableSpanEndFound=false] [tableHasHole=false] [tableID=3681]

[nongfushanquan]

/assign @hicqu

[hicqu]

Closed as duplicated with #8668 .