tikv: make requests fail quickly when the dial times out

[lysu]

What problem does this PR solve?

current impl, we need wait request-timeout(cop:60s, 2pc:20s, point-get:20s) to trigger retry.

we should fail-fast if can't connect to target host, because request-timeout thinking about processing time, but for network unreachable error, it's better to timeout quickly and retry others.

What is changed and how it works?

• add wait ready timeout again(remove by code review in the previous pr)
• fail pending request quickly if wait ready timeouted

Check List

Tests

• Unit test
• Integration test
• Manual test

random rewrite getConnArray to a unknow host.
and check can TiDB works well

Code changes

• impl

Side effects

• n/a

Related changes

• Need to cherry-pick to the release branch

Release note

• tikv: make requests fail-fast for dial timeouted

---

This change is 

[lysu]

/run-all-tests

[codecov]

Codecov Report

Merging #12819 into master will decrease coverage by 0.0449%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##            master     #12819       +/-   ##
==============================================
- Coverage   80.146%   80.1011%   -0.045%     
==============================================
  Files          465        465               
  Lines       107903     107398      -505     
==============================================
- Hits         86480      86027      -453     
  Misses       14951      14951               
+ Partials      6472       6420       -52

[sre-bot]

@coocood, @tiancaiamao, PTAL.

[jackysp]

LGTM

[coocood]

LGTM

[coocood]

Have you run the manual test again after the latest modification?

[lysu]

@coocood I have added a new failpoint to mock the wrong addr by manually:

1. build tidb with make failpoint-enable
2. start a tidb with tikv/pd with GO_FAILPOINTS_HTTP=0.0.0.0:10001 ./bin/tidb-server -store tikv -path 127.0.0.1:2379
3. inject wrong addr for once with curl -XPUT -d '1*return("182.16.5.31:8080")->return("")' http://0.0.0.0:10001/github.com/pingcap/tidb/store/tikv/injectWrongStoreAddr
4. try send other request to tidb cluster should not block, and tidb log will see this after 5s

[2019/10/22 17:16:09.210 +08:00] [INFO] [region_cache.go:504] ["switch region peer to next due to send request fail"] [current="region ID: 6, meta: id:6 start_key:\"m\" end_key:\"n\" region_epoch:<conf_ver:1 version:5 > peers:<id:7 store_id:1 > , peer: id:7 store_id:1 , addr: 182.16.5.31:8080, idx: 0"] [needReload=true] [error="context deadline exceeded"]
[2019/10/22 17:16:09.214 +08:00] [INFO] [region_cache.go:330] ["invalidate current region, because others failed on same store"] [region=10] [store=127.0.0.1:9191]
[2019/10/22 17:16:09.214 +08:00] [INFO] [region_cache.go:330] ["invalidate current region, because others failed on same store"] [region=10] [store=127.0.0.1:9191]
[2019/10/22 17:16:09.212 +08:00] [INFO] [region_cache.go:330] ["invalidate current region, because others failed on same store"] [region=12] [store=127.0.0.1:9191]

[coocood]

LGTM

[lysu]

/run-all-tests

[lysu]

/run-all-tests

[lysu]

@jackysp @coocood need a approve thx~

[sre-bot]

/run-all-tests

[sre-bot]

cherry pick to release-3.1 failed

[sre-bot]

cherry pick to release-3.0 failed