br restore failed with error “region 714153's leader's store id is 0” when pd rolling restart

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、br restore
2、pd rolling restart

2. What did you expect to see? (Required)

br restore success

3. What did you see instead (Required)

br restore failed
error="region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region; region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region; region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region; region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region"] [errorVerbose="the following errors occurred:\n - [BR:PD:ErrPDBatchScanRegion]batch scan region\n region 714153's leader's store id is 0\n github.com/pingcap/tidb/br/pkg/restore/split.CheckRegionConsistency\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:76\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:147\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:117\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n \t/workspace/source/tidb/br/pkg/restore/import.go:548\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n \t/workspace/source/tidb/br/pkg/restore/import.go:544\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n \t/workspace/source/tidb/br/pkg/restore/client.go:1458\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n \t/workspace/source/tidb/br/pkg/restore/client.go:1459\n github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \t/workspace/source/tidb/br/pkg/utils/worker.go:76\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDBatchScanRegion]batch scan region\n region 714153's leader's store id is 0\n github.com/pingcap/tidb/br/pkg/restore/split.CheckRegionConsistency\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:76\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:147\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:117\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n \t/workspace/source/tidb/br/pkg/restore/import.go:548\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n \t/workspace/source/tidb/br/pkg/restore/import.go:544\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n \t/workspace/source/tidb/br/pkg/restore/client.go:1458\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n \t/workspace/source/tidb/br/pkg/restore/client.go:1459\n github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \t/workspace/source/tidb/br/pkg/utils/worker.go:76\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDBatchScanRegion]batch scan region\n region 714153's leader's store id is 0\n github.com/pingcap/tidb/br/pkg/restore/split.CheckRegionConsistency\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:76\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:147\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:117\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n \t/workspace/source/tidb/br/pkg/restore/import.go:548\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n \t/workspace/source/tidb/br/pkg/restore/import.go:544\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n \t/workspace/source/tidb/br/pkg/restore/client.go:1458\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n \t/workspace/source/tidb/br/pkg/restore/client.go:1459\n github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \t/workspace/source/tidb/br/pkg/utils/worker.go:76\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650\n - [BR:PD:ErrPDBatchScanRegion]batch scan region\n region 714153's leader's store id is 0\n github.com/pingcap/tidb/br/pkg/restore/split.CheckRegionConsistency\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:76\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:147\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n \t/workspace/source/tidb/br/pkg/restore/split/split.go:117\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles.func1\n \t/workspace/source/tidb/br/pkg/restore/import.go:548\n github.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n \t/workspace/source/tidb/br/pkg/utils/retry.go:217\n github.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n \t/workspace/source/tidb/br/pkg/utils/retry.go:235\n github.com/pingcap/tidb/br/pkg/utils.WithRetry\n \t/workspace/source/tidb/br/pkg/utils/retry.go:216\n github.com/pingcap/tidb/br/pkg/restore.(*FileImporter).ImportSSTFiles\n \t/workspace/source/tidb/br/pkg/restore/import.go:544\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2.1\n \t/workspace/source/tidb/br/pkg/restore/client.go:1458\n github.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreSSTFiles.func2\n \t/workspace/source/tidb/br/pkg/restore/client.go:1459\n github.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n \t/workspace/source/tidb/br/pkg/utils/worker.go:76\n golang.org/x/sync/errgroup.(*Group).Go.func1\n \t/root/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n runtime.goexit\n \t/usr/local/go/src/runtime/asm_amd64.s:1650"] Error: region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region; region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region; region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region; region 714153's leader's store id is 0: [BR:PD:ErrPDBatchScanRegion]batch scan region

4. What is your TiDB version? (Required)

tidb version:
./tidb-server -V
Release Version: v7.5.2
Edition: Community
Git Commit Hash: 39ea2b3
Git Branch: HEAD
UTC Build Time: 2024-05-29 15:07:12
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-05-30T14:14:32.686+0800

[Lily2025]

/type bug
/severity major
/assign 3pointer

[Leavrth]

#51124
#50501