BR suffers a 15x performance regression when single TiKV node down

[YuJuncen]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Create a cluster with 4 or more TiKV nodes.
2. Shut down one TiKV node.
3. Execute the backup.

2. What did you expect to see? (Required)

The backup should be slightly slower than backing up a healthy cluster.

3. What did you see instead (Required)

The backup speed is about 15x slower than backing up a healthy cluster. (4 mins vs 1 hour)

4. What is your TiDB version? (Required)

Near master, but this problem is not strong relative to the version.

[YuJuncen]

The reason is:

• When we are dialing to a (down) host, it takes a long time to fail.
• BR SHOULD skip those (unreachable) stores, and BR did implemented this:
  

  tidb/br/pkg/backup/push.go

  Lines 80 to 82 in 0548d61

  	if s.GetState() != metapb.StoreState_Up {
  	logutil.CL(lctx).Warn("skip store", zap.Stringer("State", s.GetState()))
  	continue

• But in our case, it seems the returned store status is still Up even pd-ctl returns a Down status.
• Consequently, for backing up any range, we must wait about 30s for the store failed... (30s is the dial timeout) This also cannot concurrently wait for this because the mutual execution of StoreManager.
• So... If there were a store down, we may take at least 1 min for each object to be backed up. In my TPCC workload, there are 116 objects (ranges), then it costs about 1 hour to fully backup the cluster.

[YuJuncen]

BTW, the current implementation will directly jump to the fine-grained backup when there is any store unreachable:

tidb/br/pkg/backup/push.go

Lines 85 to 90 in 0548d61

	if err != nil {
	// BR should be able to backup even some of stores disconnected.
	// The regions managed by this store can be retried at fine-grained backup then.
	logutil.CL(lctx).Warn("fail to connect store, skipping", zap.Error(err))
	return nil
	}

This may slow down the backup speed in this scenario.

[YuJuncen]

There isn't trivial fix to the problem. I think we must figure out why PD will return wrong state of store firstly (Note that StoreState_Up is the default value for StoreState).

[tonyxuqqi]

StoreState_Up

@nolouch , please take a look. The question is what's the best way to tell if the store is healthy or not. Apparently StoreState_Up seems not always accurate as it may need to wait for 30 minutes to be StoreState_Down

[bufferflies]

br using client to get store information by grpc, but the pd-ctl using http interface to get it. In the http handling, PD will check the store down status:
https://github.com/tikv/pd/blob/c40e319f50822678cda71ae62ee2fd70a9cac010/server/api/store.go#L144-L151
In grpc handling, it doesn't contain this code:
https://github.com/tikv/pd/blob/c40e319f50822678cda71ae62ee2fd70a9cac010/server/api/store.go#L144-L151