changefeed stucks "etcd client outCh blocking too long, the etcdWorker may be stuck"

[fubinzh]

What did you do?

run 2 changefeed.

What did you expect to see?

changefeed lag normal

What did you see instead?

cdc changefeed stucks

[root@tc-ticdc-1 /]# tail -f /var/lib/ticdc/log/ticdc.log
[2023/05/26 12:17:23.506 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=33m57.899318088s] [role=processor]
[2023/05/26 12:17:24.505 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=33m58.899301364s] [role=processor]
[2023/05/26 12:17:25.505 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=33m59.899034374s] [role=processor]
[2023/05/26 12:17:26.506 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m0.899448022s] [role=processor]
[2023/05/26 12:17:27.505 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m1.89908253s] [role=processor]
[2023/05/26 12:17:28.505 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m2.899023389s] [role=processor]
[2023/05/26 12:17:29.506 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m3.899374028s] [role=processor]
[2023/05/26 12:17:30.505 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m4.89901759s] [role=processor]
[2023/05/26 12:17:31.506 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m5.899363364s] [role=processor]
[2023/05/26 12:17:32.505 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m6.899026892s] [role=processor]
[2023/05/26 12:17:33.598 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m7.991587301s] [role=processor]
[2023/05/26 12:17:34.505 +08:00] [WARN] [client.go:259] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=34m8.899025767s] [role=processor]

Versions of the cluster

TiCDC version (execute cdc version):

[root@tc-ticdc-1 /]# /cdc version
Release Version: v7.1.0
Git Commit Hash: f4b7ef5eef3833b26b81e9d57e76687d4ac93bc7
Git Branch: heads/refs/tags/v7.1.0
UTC Build Time: 2023-05-25 06:40:04
Go Version: go version go1.20.3 linux/amd64
Failpoint Build: false

[fubinzh]

/severity critical

[nongfushanquan]

fixed in version 6.5.3 by #9100