Delete evict leader scheduler may lead to PD crush after removing tombstone

[rleungx]

Bug Report

What did you do?

1. create a cluster with 4 TiKV using TiUP
2. add evict leader scheduler for one store, e.g. store 2
3. delete store 2 and remove tombstone by using pd-ctl
4. remove evict leader scheduler by using pd-ctl

What did you expect to see?

The evict leader scheduler is removed successfully

What did you see instead?

[2021/05/08 11:43:06.652 +08:00] [FATAL] [store.go:550] ["try to clean a store's pause state, but it is not found"] [store-id=2] [error="[PD:core:ErrStoreNotFound]store 2 not found"] [stack="github.com/pingcap/log.Fatal\n\t/go/pkg/mod/github.com/pingcap/log@v0.0.0-20210317133921-96f4fcab92a4/global.go:62\ngithub.com/tikv/pd/server/core.(*StoresInfo).ResumeLeaderTransfer\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/pd/server/core/store.go:550\ngithub.com/tikv/pd/server/core.(*BasicCluster).ResumeLeaderTransfer\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/pd/server/core/basic_cluster.go:145\ngithub.com/tikv/pd/server/cluster.(*RaftCluster).ResumeLeaderTransfer\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/pd/server/cluster/cluster.go:1100\ngithub.com/tikv/pd/server/schedulers.(*evictLeaderScheduler).Cleanup\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/pd/server/schedulers/evict_leader.go:206\ngithub.com/tikv/pd/server/cluster.(*coordinator).runScheduler\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/pd/server/cluster/coordinator.go:740"]

What version of PD are you using (pd-server -V)?

master and 5.0

[cosven]

/severity critical

[zhangjinpeng87]

This is a critical bug, we need to fix it ASAP.