import into jobs failed when kill pd leader

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、import into user_data1 from 's3://brie/mvindex-10000000-csv/test.user_data1.*.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://minio.pingcap.net:9001&force-path-style=true' WITH skip_rows=1
2、kill pd leader

2. What did you expect to see? (Required)

import into job finished

3. What did you see instead (Required)

import into job failed
select id, step, status from mysql.tidb_import_jobs where start_time >= '2023-09-13 15:47:37'
jobId: 30002, step: importing, status: failed

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.4.0-alpha
Edition: Community
Git Commit Hash: 8f8c433
Git Branch: heads/refs/tags/v7.4.0-alpha
UTC Build Time: 2023-09-13 06:50:51
GoVersion: go1.21.1
Race Enabled: false
Check Table Before Drop: false
Store: unistore

[Lily2025]

/assign D3Hunter

[D3Hunter]

the error happens when try to pause scheduler, should be catched by retry, will check it later.

tidb/br/pkg/lightning/common/retry.go

Line 107 in a7e3176

return syscallErr.Err == syscall.ECONNREFUSED || syscallErr.Err == syscall.ECONNRESET

[2023/09/13 15:49:45.280 +08:00] [INFO] [backend.go:363] ["import start"] [engineTag=`importtest`.`user_data1`:0] [engineUUID=9a09d9d7-e635-547b-86cb-e8564a7e4a95] [retryCnt=0]
[2023/09/13 15:49:45.280 +08:00] [WARN] [local.go:1481] ["fail to get region split keys and size"] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.200.73.51:2379: connect: connection refused\""]
[2023/09/13 15:49:45.281 +08:00] [INFO] [local.go:1085] ["split engine key ranges"] [engine=9a09d9d7-e635-547b-86cb-e8564a7e4a95] [totalSize=2290000000] [totalCount=10000000] [firstKey=74800000000000006E5F728000000000000001] [lastKey=74800000000000006E5F728000000000989680] [ranges=12] []
[2023/09/13 15:49:45.282 +08:00] [INFO] [local.go:1491] ["pause pd scheduler of table scope"]
[2023/09/13 15:49:45.283 +08:00] [WARN] [pd.go:993] ["failed to create or update region label rule, will try next pd address"] [error="Post \"http://tc-pd-2.tc-pd-peer.endless-ha-test-import-into-tps-2460291-1-824.svc:2379/pd/api/v1/config/region-label/rule\": dial tcp 10.200.73.51:2379: connect: connection refused"] [pdAddr=http://tc-pd-2.tc-pd-peer.endless-ha-test-import-into-tps-2460291-1-824.svc:2379]
[2023/09/13 15:49:45.284 +08:00] [WARN] [pd.go:993] ["failed to create or update region label rule, will try next pd address"] [error="Post \"http://tc-pd:2379/pd/api/v1/config/region-label/rule\": dial tcp 10.103.153.58:2379: connect: connection refused"] [pdAddr=http://tc-pd:2379]
[2023/09/13 15:49:45.284 +08:00] [ERROR] [backend.go:369] ["import failed"] [engineTag=`importtest`.`user_data1`:0] [engineUUID=9a09d9d7-e635-547b-86cb-e8564a7e4a95] [retryCnt=0] [takeTime=3.565101ms] [error="Post \"http://tc-pd:2379/pd/api/v1/config/region-label/rule\": dial tcp 10.103.153.58:2379: connect: connection refused"]