two pods are not online at the same time during tikv rolling restart, which cause service unavailable

[Lily2025]

Bug Report

What version of Kubernetes are you using?
Client Version: v1.31.1
Kustomize Version: v5.4.2

What version of TiDB Operator are you using?
v2

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
db-174778593390501-tidb-6w70yg 3/3 Running 0 60m
db-174778593390501-tidb-ypbm4q 3/3 Running 0 62m
db-174778593390501-worker-tidb-pq59rs 3/3 Running 0 57m
db-174778593390502-tidb-6qsns5 3/3 Running 0 52m
db-174778593390502-tidb-g4bfu6 3/3 Running 0 53m
db-174778593390502-worker-tidb-v8qxj5 3/3 Running 0 48m
db-174778593390503-tidb-223ux6 3/3 Running 0 43m
db-174778593390503-tidb-hz4is4 3/3 Running 1 (43m ago) 45m
db-174778593390503-worker-tidb-1vr4bi 3/3 Running 0 39m
db-174780869947901-tidb-vxfonl 3/3 Running 71 (47h ago) 7d15h
db-174780869947901-worker-tidb-tkc77f 3/3 Running 44 (47h ago) 7d13h
db-a576e8f4-compute-tiflash-autwzi 3/3 Running 0 46h
db-a576e8f4-compute-tiflash-l7y93w 3/3 Running 0 46h
db-a576e8f4-coprocessor-worker-69d897bb5-gt5wq 1/1 Running 23 (15h ago) 8d
db-a576e8f4-coprocessor-worker-69d897bb5-z4hdb 1/1 Running 7 (15h ago) 46h
db-a576e8f4-pd-21ldog 1/1 Running 11 (23h ago) 5d20h
db-a576e8f4-pd-35iy2c 1/1 Running 56 (21h ago) 5d20h
db-a576e8f4-pd-f4yjb1 1/1 Running 5 (2d17h ago) 2d17h
db-a576e8f4-scheduler-ih22he 1/1 Running 13 (19h ago) 46h
db-a576e8f4-scheduler-jybqkh 1/1 Running 9 (21h ago) 46h
db-a576e8f4-tikv-1yorcy 1/1 Running 0 73m
db-a576e8f4-tikv-iiqaes 1/1 Running 0 72m
db-a576e8f4-tikv-l8y5cy 1/1 Running 0 70m
db-a576e8f4-tikv-uqekoh 1/1 Running 0 68m
db-a576e8f4-tikv-v7tmrw 1/1 Running 0 66m
db-a576e8f4-tikv-vhbk1b 1/1 Running 0 65m
db-a576e8f4-tikv-worker-5b66c8764c-78ggz 1/1 Running 11 (2d1h ago) 8d
db-a576e8f4-tikv-worker-5b66c8764c-xzwmg 1/1 Running 19 (15h ago) 8d
db-a576e8f4-tso-gbnsro 1/1 Running 13 (21h ago) 2d16h
db-a576e8f4-tso-zskdim 1/1 Running 13 (20h ago) 2d16h
db-a576e8f4-write-tiflash-6gyaxe 3/3 Running 0 46h
db-a576e8f4-write-tiflash-ndqrx6 3/3 Running 0 46h

What's the status of the TiDB cluster pods?
normal

What did you do?
1、run workload
2、scale down tikv

What did you expect to see?
no unexpected error

What did you see instead?
1、workload report Error 9005 (HY000): Region is unavailable
2、two pods are not online at the same time during rolling restart

[Lily2025]

/type bug
/assign liubog2008

[Lily2025]

/v2

[Lily2025]

/label v2

[ti-chi-bot]

@Lily2025: The label(s) v2 cannot be applied. These labels are supported: help wanted, good first issue.

Details

In response to this:

/label v2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.