You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Inject 1000ms network delay in all TiDB nodes to 192.168.80.2, which deploy TiKV 192.168.80.2:23360 and 192.168.80.2:23361
[root@192.168.78.2]# ping 192.168.80.2
PING 192.168.80.2 (192.168.80.2) 56(84) bytes of data.
64 bytes from 192.168.80.2: icmp_seq=1 ttl=62 time=546 ms
64 bytes from 192.168.80.2: icmp_seq=2 ttl=62 time=217 ms
64 bytes from 192.168.80.2: icmp_seq=3 ttl=62 time=145 ms
64 bytes from 192.168.80.2: icmp_seq=4 ttl=62 time=500 ms
64 bytes from 192.168.80.2: icmp_seq=5 ttl=62 time=1091 ms
[root@192.168.78.2]# ping 192.168.80.3
PING 192.168.80.3 (192.168.80.3) 56(84) bytes of data.
64 bytes from 192.168.80.3: icmp_seq=1 ttl=62 time=0.086 ms
64 bytes from 192.168.80.3: icmp_seq=2 ttl=62 time=0.079 ms
64 bytes from 192.168.80.3: icmp_seq=3 ttl=62 time=0.082 ms
[root@192.168.78.2]# ping 192.168.80.4
PING 192.168.80.4 (192.168.80.4) 56(84) bytes of data.
64 bytes from 192.168.80.4: icmp_seq=1 ttl=62 time=0.088 ms
64 bytes from 192.168.80.4: icmp_seq=2 ttl=62 time=0.072 ms
64 bytes from 192.168.80.4: icmp_seq=3 ttl=62 time=0.058 ms
192.168.78.3 and 192.168.78.4 are same with upper too.
Then I add some metrics ( #973 ) in batch-client, this is the root cause to make Insert statement execute too slowly.
In the following metric, since I inject around 1 second network latency between 192.168.78.2 and 192.168.80.2, so the max batch conn send duration is 15s.
In the following metric, the max got-resp is 1.14min, and max wait-to-send is 16s, this is unexpected.
The text was updated successfully, but these errors were encountered:
I found this issue in #970
Topology
Version: nightly
Inject network latency
Inject 1000ms network delay in all TiDB nodes to
192.168.80.2
, which deploy TiKV192.168.80.2:23360
and192.168.80.2:23361
192.168.78.3
and192.168.78.4
are same with upper too.Test
Load: stale-read request
tidb_kv_read_timeout: 100
max_execution_time: 400
And run more load in TiDB
192.168.78.2
:The following are related metrics, some of metrics here are not as expected:
DataNotReady
backoff.DataNotReady
is cause by lock. Following is TiKV log:444257405948395529
from upper log to search in TiDB slow query, which is a TiDB internal query, from statistic.the related slow log is following:
The slow query execution plan is following:
Then I add some metrics ( #973 ) in batch-client, this is the root cause to make Insert statement execute too slowly.
In the following metric, since I inject around 1 second network latency between
192.168.78.2
and192.168.80.2
, so the maxbatch conn send duration
is 15s.In the following metric, the max
got-resp
is 1.14min, and maxwait-to-send
is 16s, this is unexpected.The text was updated successfully, but these errors were encountered: