BR restore failed: Corruption: block checksum mismatch

[hi-rustin]

Bug Report

What version of TiKV are you using?

MySQL [(none)]> SELECT type AS component, group_concat(DISTINCT concat(version, ':', git_hash)) AS versions FROM information_schema.cluster_info GROUP BY type;
+-----------+--------------------------------------------------------------------+
| component | versions                                                           |
+-----------+--------------------------------------------------------------------+
| tidb      | 8.2.0-alpha-158-g6aef624:6aef624c8d4fc6e0301c4ef2f527f3b8a056edd7  |
| pd        | 8.2.0-alpha-28-g2683373:2683373d77c12adad19c9875d984d72f6e504869   |
| tikv      | 8.2.0-alpha:86e0ec3006dd306a254481a3631069370b3a0537               |
| tiflash   | 8.2.0-alpha-27-gd1d990fb8:d1d990fb8058b49c01e140c1580a400144e5a167 |
+-----------+--------------------------------------------------------------------+
4 rows in set (0.09 sec)

What operating system and CPU are you using?

Steps to reproduce

1. Create a tidb cluster: tiup playground nightly
2. Use br to restore the data: tiup br restore db --pd "127.0.0.1:2379" --db sbtest1 --storage "s3://qe-testing/kernel-testing/x/testdata/sbtest500w10t/?access-key=x&secret-access-key=x&endpoint=http%3a%x%2fks3-cn-beijing-internal.ksyuncs.com&force-path-style=false&region=Beijing&provider=ks"

What did you expect?

Restire success.

What did happened?

🎉 TiDB Playground Cluster is started, enjoy!

Connect TiDB:    mysql --comments --host 127.0.0.1 --port 4000 -u root
TiDB Dashboard:  http://127.0.0.1:2379/dashboard
Grafana:         http://127.0.0.1:3000
tikv quit: signal: killed
[2024/05/15 16:30:45.522 +08:00] [WARN] [client.rs:155] ["failed to update PD client"] [error="Other(\"[components/pd_client/src/util.rs:377]: cancel reconnection due to too small interval\")"] [thread_id=12]
[2024/05/15 16:30:45.553 +08:00] [WARN] [pd.rs:1773] ["report min resolved_ts failed"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [thread_id=31]
[2024/05/15 16:30:46.416 +08:00] [ERROR] [util.rs:721] ["connect failed"] [error="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [endpoints=http://127.0.0.1:2379] [thread_id=57]
[2024/05/15 16:30:46.560 +08:00] [WARN] [client.rs:155] ["failed to update PD client"] [error="Other(\"[components/pd_client/src/util.rs:377]: cancel reconnection due to too small interval\")"] [thread_id=12]
[2024/05/15 16:30:46.583 +08:00] [ERROR] [sst_importer.rs:414] ["download failed"] [err_code=KV:Engine:Engine] [err="Engine Engine(Status { code: IoError, sub_code: None, sev: NoError, state: \"Corruption: block checksum mismatch: stored = 2324967102, computed = 1400147483, type = 1  in /root/.tiup/data/UCpVa5x/tikv-0/data/import/.temp/17002fa1-f7a4-4f5e-bb02-ef8da640337c_223_1_176_write_2.sst offset 2286678 size 13219\" })"] [name=1/1859_137_05001ea11fcf2296f41bba35be3f29e4900abb14e3a19292848e50b6b4fc658e_1684460809036_write.sst] [meta="uuid: 17002FA1F7A44F5EBB02EF8DA640337C range { start: 7480000000000000E55F7280000000000CCC2F end: 7480000000000000E55F728000000000133898 } length: 40184695 cf_name: \"write\" region_id: 223 region_epoch { conf_ver: 1 version: 176 } cipher_iv: 53177B8C30D827585B05ED9322CA87C6"] [thread_id=103]
[2024/05/15 16:30:46.653 +08:00] [INFO] [util.rs:639] ["connecting to PD endpoint"] [endpoints=http://127.0.0.1:2379] [thread_id=12]
[2024/05/15 16:30:46.817 +08:00] [ERROR] [sst_importer.rs:414] ["download failed"] [err_code=KV:Engine:Engine] [err="Engine Engine(Status { code: IoError, sub_code: None, sev: NoError, state: \"Corruption: block checksum mismatch: stored = 2324967102, computed = 1271725483, type = 1  in /root/.tiup/data/UCpVa5x/tikv-0/data/import/.temp/05c1513b-6395-4160-9104-ac3298d673de_147_1_176_write_2.sst offset 290551 size 13191\" })"] [name=1/1825_135_174da67457dcfc8166b58f064e0919fbf3c8596b576bf8201a71e1293af28a81_1684460792641_write.sst] [meta="uuid: 05C1513B639541609104AC3298D673DE range { start: 7480000000000000E15F72 end: 7480000000000000E15F728000000000066DA8 } length: 40180624 cf_name: \"write\" region_id: 147 region_epoch { conf_ver: 1 version: 176 } cipher_iv: 2A395D92A273E2AE329AB3F64D65E3D0"] [thread_id=99]
[2024/05/15 16:30:46.820 +08:00] [ERROR] [util.rs:497] ["request failed, retry"] [err_code=KV:Pd:Grpc] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [thread_id=57]
[2024/05/15 16:30:46.820 +08:00] [WARN] [pd.rs:1773] ["report min resolved_ts failed"] [err="Grpc(RpcFailure(RpcStatus { code: 4-DEADLINE_EXCEEDED, message: \"Deadline Exceeded\", details: [] }))"] [thread_id=31]
[2024/05/15 16:30:46.904 +08:00] [WARN] [sst_service.rs:925] ["send rpc response"] [err=RemoteStopped] [thread_id=99]
...
check detail log from: /root/.tiup/data/UCpVa5x/tikv-0/tikv.log

Detail BR log in /tmp/br.log.2024-05-15T17.15.07+0800 
DataBase Restore <-----------------------------------------------------------------------|..............................................................................> 48.00%{"level":"warn","ts":"2024-05-15T17:15:23.244612+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00096b500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
DataBase Restore <-----------------------------------------------------------------------|..............................................................................> 48.00%