backup: --ratelimit doesn't work as intended

[YuJuncen]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.
   Backup with --ratelimit 8 in a cluster with one TiKV.

2. What did you expect to see?
   The speed should less than 8MB/s.

3. What did you see instead?
   The speed is 28.30MB/s. And the table below shows the relation among ratelimit, concurrency and real speed.
   Workload: TPCC 10 warehouses(about 1GB).

ratelimit	concurrency	speed(MB/s)
32	4	55.66
32	1	31.28
8	4	28.30
8	1	14.24
4	4	16.94
4	1	8.37
2	4	9.30
2	1	4.50

4. What version of BR and TiDB/TiKV/PD are you using?
   tiup v5.0.0

[kennytm]

The documentation states that

--ratelimit: specifies the maximum speed at which a backup operation is performed (MiB/s) on each TiKV node.

So we either fix TiKV or the doc.

[3pointer]

If we set ratelimit to ratelimit/concurrency, can we solve this problem?🤔️

[kennytm]

it is solved only if the ranges are distributed evenly, otherwise we'll get a slow long-tail.

[3pointer]

it is solved only if the ranges are distributed evenly, otherwise we'll get a slow long-tail.

Then how about we set concurrency to 1, if we found ratelimit not nil? @kennytm

[ryangao0513]

v4.0.14
A customer wants to limit the speed of a single tikv node to 240MB/s, but because the concurrency is forced to be 1, the speed cannot go up. A single tikv node is less than 20MB/s. It takes 18h to back up three 22T copies of the database, while mydumper only takes 10h. So is it better not to force concurrency to 1, but to set ratelimit / concurrency as the limit?