client: support add ready for resp and add backoff mechanism

[HuSharp]

What problem does this PR solve?

Issue Number: ref #6556

What is changed and how does it work?

PR Summary

1. Add ready for resp

• Have goroutine reconnectMemberLoop call updateMember periodically. When calling ScheduleCheckMemberChanged channel, we need to wait for the goroutine to update members until ready or timeout.

2. Add backoff mechanism

• When waiting for the goroutine to update, the expo function can be used to backoff to sleep when an error is encountered.

Reproduce Step

1. enable fail point, like gRPC is throttling, cannot read from etcd.
   curl -X PUT -d 'return(10)' http://tc-pd-1.tc-pd-peer.csn-simulator-big-cluster-vd62g.svc:2379/pd/api/v1/fail/github.com/tikv/pd/pkg/etcdutil/SlowEtcdKVGet

2. simulate pd lost leader
   curl -X PUT -d 'return("2346857576170797299")' http://tc-pd-1.tc-pd-peer.csn-simulator-big-cluster-vd62g.svc:2379/pd/api/v1/fail/github.com/tikv/pd/server/exitCampaignLeader

Reproduce Result

Grpc request GetMember keeps high:

TiKV side show

PR Effect

The Grpc GetMember call was reduced from 778 to 216, which is relative to the TiDB numbers and client requests for triaging checkLeader.

And more tests are necessary to ensure that no further issues arise.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

Release note

None.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[codecov]

Codecov Report

Merging #6974 (670bc4a) into master (1743552) will decrease coverage by 0.02%.
Report is 1 commits behind head on master.
The diff coverage is 74.34%.

❗ Current head 670bc4a differs from pull request most recent head 1005871. Consider uploading reports for the commit 1005871 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6974      +/-   ##
==========================================
- Coverage   74.25%   74.23%   -0.02%     
==========================================
  Files         433      435       +2     
  Lines       45904    46010     +106     
==========================================
+ Hits        34085    34157      +72     
- Misses       8810     8842      +32     
- Partials     3009     3011       +2     

Flag	Coverage Δ
unittests	74.23% <74.34%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[niubell]

Use a new pr #6978 (release-6.5), #6995 (master) to add backoff mechanism and this pr should be closed

[HuSharp]

Use a new pr #6978 (release-6.5), #6995 (master) to add backoff mechanism and this pr should be closed

Closed.