ticdc lag reached more than 10min and ticdc crash when inject pdleader data io hang

[Lily2025]

What did you do?

1、run tpcc with threads 10 and warehouse 1000
2、After 10 minutes, simulates the io of pd leader is hang but the pod is still active
fault start time：2023-05-15 12:05:38
3、After 10 minutes, recovery the fault
fault recover time：2023-05-15 12:15:38

What did you expect to see?

changefeed lag is less than 30s

What did you see instead?

1 、changefeed lag reached more than 10min after inject fault
2、 ticdc crash

Versions of the cluster

git hash：1335f98cdbaf77239bbcbc6b61561e4254449ffe

current status of DM cluster (execute query-status <task-name> in dmctl)

No response

[Lily2025]

/remove-area dm
/area ticdc

[Lily2025]

/severity major

[asddongmen]

#9106 alleviate this problem, in testing, it was found that after merging this PR, there is only a 50% chance of encountering the cdc stuck issue.

[nongfushanquan]

/assign @fubinzh

[zhangjinpeng87]

What is the root cause of the lag since TiCDC doesn't depends on PD leader's IO, TiCDC just asynchronously push the checkpoint to PD. cc @nongfushanquan

[fubinzh]

This issue still seen with v7.2.0
chaos injection time: 2023-06-21 19:14 - 19:19

[nongfushanquan]

TiCDC nodes have to keep connection with the PD , but in this scenario , PD can't read the leader's information from etcd , which may be caused by the following issue
etcd-io/etcd#12528.

[asddongmen]

It's a etcd issue, we can't fix now.

[Lily2025]

ticdc crash when inject pd leader io delay 1s
[2023/10/18 17:36:14.719 +08:00] [INFO] [chaos.go:64] ["Run chaos"] [name=iochaos_io_delay] [selectors="[endless-ha-test-ticdc-tps-3240159-1-638/tc-pd-2]"] [selectorsRetainPolicy(selectors)="[endless-ha-test-ticdc-tps-3240159-1-638/tc-pd-2]"] [targetSelectors="[nil]"] [targetSelectorsRetainPolicy(targetSelectors)="[nil]"] [experimentSpec="IODelaySpec{Duration: "", Scheduler: , Delay: "1s", Path: "/var/lib/pd/data/**/*", Percent: 100}"]

[nongfushanquan]

/remove-label affects-7.5

[ti-chi-bot]

@nongfushanquan: The label(s) affect-7.5 cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, ok-to-test, needs-ok-to-test, affects-5.3, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, may-affects-5.3, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, question, release-blocker, wontfix.

In response to this:

/remove-label affect-7.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[asddongmen]

If PD is able to upgrade its etcd version to 3.4.31, it's likely that the issue will be resolved.
Ref: etcd-io/etcd#17465 (comment)
cc @flowbehappy

[asddongmen]

After the merge of #10881, the checkpointTs lag during pd-leader-io-hang cases was reduced to less than 120s, meeting the requirement.

cc @Lily2025