v6.5.1 lightning import data to master fail for pd reset the connection

[seiya-annie]

Bug Report

What did you do?

tidb cluster upgrade from 6.0.0 to 6.5.1-pre
import data to 6.0.0 cluster using 6.5.1 lightning

What did you expect to see?

import success

What did you see instead?

import fail
print many warning:
{"level":"warn","ts":"2023-03-07T03:08:03.956Z","logger":"etcd-client","caller":"v3@v3.5.2/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c22380/src-tidb-pd.fb-testbed-comp-2nrpj:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: Error while dialing dial tcp 10.244.7.55:2379: connect: connection refused""}

What version of PD are you using (pd-server -V)?

v6.5.1-pre

[seiya-annie]

lightning.log.2023-03-06T16.54.59Z.gz

[seiya-annie]

pd-downstream.log.gz
pd-upstream.log.gz

[seiya-annie]

pingcap/tidb#41973

[bufferflies]

the restart related log :

[2023/03/06 16:56:27.610 +00:00] [INFO] [region.go:679] ["region Version changed"] [region-id=8021] [detail="StartKey Changed:{7480000000000007FFA300000000000000F8} -> {7480000000000007FFA35F728000000000FF27AC410000000000FA}, EndKey:{748000FFFFFFFFFFFFF900000000000000F8}"] [old-version=1056] [new-version=1059]
[2023/03/06 16:56:44.671 +00:00] [INFO] [versioninfo.go:89] ["Welcome to Placement Driver (PD)"]
[2023/03/06 16:56:44.671 +00:00] [INFO] [versioninfo.go:90] [PD] [release-version=v6.7.0-alpha]

[bufferflies]

upstream version：

branch: master
git-hash=bc5234cf4fc81050ed8352aa19293f088798491b]

down stream:

branch: release-6.5
git-hash= 51f382c73ee7bff691874cd44b07ca0a51e2dc73

[nolouch]

panic

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1e36ed8]

goroutine 3556 [running]:
github.com/tikv/pd/server/schedule.allowLeader(0x2958800?, 0xc00262c1e0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/region_scatterer.go:410 +0x58
github.com/tikv/pd/server/schedule.(*RegionScatterer).scatterRegion.func1(0x2956e20?, {{0xc00185ca70, 0x2, 0x2}, 0xc0005f8aa0, 0xc0005f8ac0})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/region_scatterer.go:348 +0x40e
github.com/tikv/pd/server/schedule.(*RegionScatterer).scatterRegion(0xc00087c680, 0xc0023e0700, {0x0, 0x0})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/region_scatterer.go:357 +0x6db
github.com/tikv/pd/server/schedule.(*RegionScatterer).Scatter(0xc00087c680, 0xc0023e0700, {0x0, 0x0})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/schedule/region_scatterer.go:301 +0x228
github.com/tikv/pd/server.(*GrpcServer).ScatterRegion(0xc0015080d0, {0x39e53f8?, 0xc00262c0f0?}, 0xc0006679d0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/grpc_service.go:1336 +0x365
github.com/pingcap/kvproto/pkg/pdpb._PD_ScatterRegion_Handler.func1({0x39e53f8, 0xc00262c0f0}, {0x2baa840?, 0xc0006679d0})
	

the root cause：

pd/server/schedule/region_scatterer.go

Lines 410 to 413 in c40e319

	rule := fit.GetRuleFit(peer.GetId()).Rule
	if rule.IsWitness {
	return false
	}

the peer maybe not in the rule.