br restore encrypted backup failed "Bad table magic number"

[fubinzh]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Prepare test data in TiDB
2. Run br backup and restore, specify crypter.method and crypter.key

/br  backup  db "-s" "s3://tmp/br-backup-basic5f2db403-d9c1-43ba-8fd2-4708d4a93e29?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true" "-u" "http://upstream-pd.brie-acceptance--tps-540378-1-915:2379" "--db" "IFRS9" "--use-backupmeta-v2" "--crypter.method" "aes128-ctr" "--crypter.key" "0123456789abcdef0123456789abcdef"

/br  restore  db "-s" "s3://tmp/br-backup-basic5f2db403-d9c1-43ba-8fd2-4708d4a93e29?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true" "-u" "http://downstream-pd.brie-acceptance--tps-540378-1-915:2379" "--db" "IFRS9" "--crypter.method" "aes128-ctr" "--crypter.key" "0123456789abcdef0123456789abcdef"

2. What did you expect to see? (Required)

br restore should succeed.

3. What did you see instead (Required)

br restore fails.

br log:
[BR:KV:ErrKVDownloadFailed]download sst failed; Engine Engine("Corruption: Bad table magic number: expected 9863518390377041911, found 2044396524680513383 in /var/lib/tikv/data/import/.temp/f4060c3b-e897-44ff-9eb6-59d960a480a2_1254_1_628_write.sst"): [BR:KV:ErrKVDownloadFailed]download sst failed;

Full log:
tests https://tcms.pingcap.net/dashboard/executions/plan/540378 on testbed brie-acceptance--tps-540378-1-915: Failure
artifacts:
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/prometheus.tar
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/upstream-pd-0/pd.log
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/upstream-tidb-0/tidb.log
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/upstream-tikv-0/tikv.log
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/prometheus.tar
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/downstream-pd-0/pd.log
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/downstream-tidb-0/tidb.log
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/downstream-tikv-0/tikv.log
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/lightning/tmp.lightning.log*.tar
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/lightning2/tmp.lightning.log*.tar
http://minio.pingcap.net:9000/minio/test-infra-testground/archive/brie-acceptance--tps-540378-1-915/br/tmp.br.log*.tar

4. What is your TiDB version? (Required)

[release-version=v5.4.0-nightly]
[git-hash=4f19445b9dbfcb0a8dd1421401ec38f67b787a8b]
[git-branch=heads/refs/tags/v5.4.0-nightly]
[go-version=go1.16.4]
[utc-build-time="2021-11-28 04:07:36"]

[fubinzh]

This issue only occurs in master build, for v5.3.0, it is OK.

[joccau]

• This bug is imported in PR(encryption: support seek for encryption writer tikv/tikv#10929)
• It causes getting a bad SST after decrypt when download SST
• Enter a incorrect buffer length when decrypt buffer from reading SST

[fubinzh]

verified, close the issue.

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.