backoff pd api when fails

[hihihuhu]

Enhancement Task

particularly, like pd get member request should have a backoff, otherwise it could overload pd and prevent it recover from some temporary issues
like for v6.5.1, https://github.com/tikv/pd/blob/v6.5.1/client/base_client.go#L306

[nolouch]

This updateMember use a memberUpdateInterval(1 min), I think we can see it as a backoff mechanism.

[nolouch]

and for tikv's pd client, @rleungx has increased the retry interval. tikv/tikv#14954. if not works, we may need consider backoff and increase the max retry time. cc @rleungx

[hihihuhu]

@nolouch thanks for the reply, the behavior we observe it there are excessive, a few thousand qps, getMember calls from tidb components to pd when pd leader is already having issue

in tidb, it could also actively triggers CheckLeader in additional to the periodical one https://github.com/tikv/pd/blob/v6.5.1/client/base_client.go#L135C2-L135C2. for exmaple, it looks like when tso allocation fails it would schedule a CheckLeader

i would be great to reduce has some backoff for this particular scenario, because the pd leader is already having issue at that point, any further load could make thing worse

[nolouch]

Got, On the TiDB side, all requests should already have a backoff mechanism via client-go's backoff but some paths may not be covered. like you said the CheckLeader. We will sort out the calling side and do some optimization.

[nolouch]

BTW, tikv/tikv#13673, in pd-client v2 , we do not need to retry in the inner of the client, so we no need backoff to reduce the requests. then this problem can be significantly improved.

[nolouch]

TiDB Side

#6978 try to reduce the GetMemeber Request. we can see the preliminary test results:

The RPC call was reduced from 3.22k to 170 ops, which is relative to the TiDB numbers and client requests for triaging checkLeader. This reduction could be more significant in larger cluster scenarios.

And more tests are necessary to ensure that no further issues arise.

[nolouch]

TiKV Side

tikv/tikv#15429 try to reduce the retries in tikv side(all tidb is no workload), details test can see in PR, the result like:

Before

After