pd can not select new leader which lead qps drop to zero when inject pdleader io delay 500ms last for 5mins

[Lily2025]

Bug Report

What did you do?

1、run workload
2、inject pdleader io delay 500ms last for 5mins
apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
name: kv-timeout-data
namespace: testbed-xxx
spec:
action: latency
mode: one
selector:
namespaces:
- testbed-xxx
labelSelectors:
statefulset.kubernetes.io/pod-name: tc-pd-0
volumePath: /var/lib/pd
path: "/var/lib/pd/data/**/*"
delay: "500ms"
percent: 100
duration: "300s"

What did you expect to see?

1、pd can select new leader when inject pdleader io delay 500ms last for 5mins
2、qps can recover within 2mins when inject pdleader io delay 500ms

What did you see instead?

pd can not select new leader when inject pdleader io delay 500ms last for 5mins
write qps drop to zero

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v6.5.0-nightly
Edition: Community
Git Commit Hash: 77d6f5b
Git Branch: heads/refs/tags/v6.5.0-nightly
UTC Build Time: 2023-10-07 10:21:09
2023-10-19T09:33:53.650+0800

[Lily2025]

/type bug
/severity major
/assign HuSharp

[HuSharp]

Situation

It looks like pd0 kept trying to elect leader timed out for five minutes. Finally, pd2 is elected.

• It looks like pd0 is running for leader because the apply is slow (io delay).
• etcd leader was updated at the end of pd0's campaign.
• etcd updated leader because heartbeat is slow.
• The qps drop to zero continue time depends on how long it took for etcd to change leader.

Solution

maybe we need to maintain a slice record election times

[Lily2025]

/open

[Lily2025]

The phenomenon still exists

[JmPotato]

Fixed by #7737.