lightning parallel import failed with continuous injection of PD failures during import process

[seiya-annie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. two lightning parallel import 70G data
2. during import, continuous injection of PD failures

2. What did you expect to see? (Required)

lightning import successfully

3. What did you see instead (Required)

[2023/04/25 07:52:18.811 +00:00] [ERROR] [restore.go:1575] ["failed to import table"] [table=`IFRS9`.`ECL_RETAIL_LOAN_BIZ_INFO`] [error="Post \"http://tc-pd.lightning-ha-test/
[2023/04/25 07:52:18.811 +00:00] [ERROR] [restore.go:1579] ["restore table failed"] [table=`IFRS9`.`ECL_RETAIL_LOAN_BIZ_INFO`] [takeTime=2h13m53.212184129s] [error="[Lightnin
[2023/04/25 07:52:18.811 +00:00] [INFO] [restore.go:1322] ["cancel periodic actions"] [do=true]                                                                               
[2023/04/25 07:52:18.811 +00:00] [INFO] [restore.go:1885] ["switch import mode"] [mode=Normal]                                                                                
[2023/04/25 07:52:18.835 +00:00] [ERROR] [restore.go:1541] ["restore all tables data failed"] [takeTime=2h13m53.243169911s] [error="[Lightning:Restore:ErrRestoreTable]restore
[2023/04/25 07:52:18.835 +00:00] [ERROR] [restore.go:479] ["run failed"] [step=4] [error="[Lightning:Restore:ErrRestoreTable]restore table `IFRS9`.`ECL_RETAIL_LOAN_BIZ_INFO` 
[2023/04/25 07:52:18.835 +00:00] [INFO] [restore.go:1184] ["everything imported, stopping periodic actions"]                                                                  
[2023/04/25 07:52:18.835 +00:00] [ERROR] [restore.go:489] ["the whole procedure failed"] [takeTime=2h13m54.351055139s] [error="[Lightning:Restore:ErrRestoreTable]restore tabl
[2023/04/25 07:52:18.883 +00:00] [ERROR] [main.go:103] ["tidb lightning encountered error stack info"] [error="[Lightning:Restore:ErrRestoreTable]restore table `IFRS9`.`ECL_R

4. What is your TiDB version? (Required)

/ # ./tidb-lightning -V
Release Version: v6.5.2-20230424
Git Commit Hash: 9183ddc
Git Branch: heads/refs/tags/v6.5.2-20230424
Go Version: go1.19.8
UTC Build Time: 2023-04-24 10:15:49
Race Enabled: false

[seiya-annie]

lightning.log.2023-04-25T05.38.22Z.gz