BR download / import give up retry when meeting retryable gRPC errors

[YuJuncen]

Bug Report

1. Minimal reproduce step (Required)

Inject a fail point at here, by status.Error

tidb/br/pkg/restore/import.go

Lines 315 to 319 in 086a63f

	failpoint.Inject("restore-storage-error", func(val failpoint.Value) {
	msg := val.(string)
	log.Debug("failpoint restore-storage-error injected.", zap.String("msg", msg))
	e = errors.Annotate(e, msg)
	})

for example:

failpoint.Inject("restore-gRPC-error", func(val failpoint.Value) {
	e = status.Error(codes.Unavailable, "the connection to TiKV has been cut by a neko, meow :3")
})

Then execute restore with GO_FAILPOINTS=github.com/pingcap/tidb/br/pkg/restore/restore-gRPC-error=1*return(true).

2. What did you expect to see? (Required)

The error should be retried, due to

tidb/br/pkg/restore/backoff.go

Lines 68 to 71 in 086a63f

	switch status.Code(err) {
	case codes.Unavailable, codes.Aborted:
	bo.delayTime = 2 * bo.delayTime
	bo.attempt--

3. What did you see instead (Required)

[2021/08/20 13:39:43.652 +08:00] [WARN] [backoff.go:76] ["unexcepted error, stop to retry"] [error="rpc error: code = Unavailable desc = the connection to TiKV has been cut by a neko, meow :3"]

4. What is your TiDB version? (Required)

Release Version: v5.0.0-master
Git Commit Hash: 6cb0c4abc210eb45c762b5d5daf561f3f77a13f9
Git Branch: master
Go Version: go1.16.4
UTC Build Time: 2021-06-02 01:33:28
Race Enabled: false

[YuJuncen]

/component br
/sig migrate

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions