Changefeed lag keeps increasing after a changefeed restart

[zhaoxinyu]

What did you do?

1. Create a changefeed with storage sink.
2. The downstream storage system (e.g. minio) experiences a temporary unavailability.

What did you expect to see?

Changefeed is restarted and the tables are replicated normally.

What did you see instead?

Changefeed is restarted, but some of the tables are not being replicated normally. And a large amount of logs are printed as follows:

[2023/02/12 01:39:38.441 +08:00] [WARN] [replication_set.go:741] ["schedulerv3: ignore input, unexpected replication set state"] [tableState="table_id:36489 span:<{table_id:36489,start_key:748000000000008eff895f720000000000fa,end_key:748000000000008eff895f730000000000fa}> state:Absent checkpoint:<> stats:<> "] [captureID=ee34951b-2232-4afc-a557-73349adea798] [replicationSet="{\"Changefeed\":{\"Namespace\":\"default\",\"ID\":\"longrun-s3-test-02\"},\"Span\":\"{table_id:36489,start_key:748000000000008eff895f720000000000fa,end_key:748000000000008eff895f730000000000fa}\",\"State\":\"Replicating\",\"Primary\":\"ee34951b-2232-4afc-a557-73349adea798\",\"Captures\":{\"ee34951b-2232-4afc-a557-73349adea798\":\"Primary\"},\"Checkpoint\":{\"checkpoint_ts\":439387383142023858,\"resolved_ts\":439387383194452934},\"Stats\":{\"stage_checkpoints\":null}}"]

Versions of the cluster

TiCDC version (execute cdc version):

v6.6.0

[zhaoxinyu]

/assign @overvenus

[overvenus]

See more #7657