retry short read reqs immediately on server busy #990
Conversation
Signed-off-by: zyguan <zhongyangguan@gmail.com>
Signed-off-by: zyguan <zhongyangguan@gmail.com>
Signed-off-by: zyguan <zhongyangguan@gmail.com>
Signed-off-by: zyguan <zhongyangguan@gmail.com>
|
/cc @crazycs520 |
|
|
| return bo.Backoff(args.cfg, args.err) | ||
| } | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Could you explain the purpose of these code? I didn't understand it 🤔
There was a problem hiding this comment.
It's used to maintain delayed backoffs.
- "
pendingBackoffs[addr]is not nil" means there is a delayed backoff which should be applied on retrying the corresponding store. delayBoTiKVServerBusyis the callback for recording delayed backoffs.backoffOnRetryis used to apply pending backoffs on retry.- when there is no candidate, the kv client just return a fake error and let the caller do retry. to avoid potential frequent rpcs, we call
backoffOnFakeErrbefore returning a fake error, which chooses a pending backoff with the largest base duration and applies it.
There was a problem hiding this comment.
Can this be done by adding "busy" flag to the stores, instead of maintaining maps here?
There was a problem hiding this comment.
If you mean the store struct in region cache, probably no I think. They have different scope/lifetime, that is, each SendReqCtx call should has its own pendingBackoffs, and has no effect to each others. Maybe we can add a field to the replica struct, however, this PR is just a quick-and-dirty fix to the customer issue, I just try to make the least change.
MyonKeminta
dismissed
their stale review
Wait for @crazycs520 's new solution
If a read-only workload running with a short max-execution-time (eg. 500ms), injecting IO delay faults will cause the QPS amost drop to zero, because the queries will be interrupted during server-busy backoff and have no chance to try other available peers.
To address the issue, let the sender retry those requests immediately and do backoff lazily. Here is the test results of injecting IO delay (1s) on one tikv for 3 minutes.
