Removed tombstone stores show again if transfer pd leader during scale in

[mayjiang0203]

Bug Report

What did you do?

Test steps:

1. Offline one store name as StoreA;
2. During the offline, transfer pd leader from A to B;
3. After offline finished, use pd-ctl delete tombstone nodes;
4. Tansfer PD leader from B to A, check stores list.

What did you expect to see?

In step 4, no StoreA show.

What did you see instead?

StoreA show as disconnected as below.
{
"store": {
"id": 24439739,
"address": "tc-tikv-3.tc-tikv-peer.testbed-oltp-hm-fv98h.svc:20160",
"version": "6.0.0-alpha",
"status_address": "0.0.0.0:20180",
"git_hash": "4476e273804abe376b5fbcf0a7cbd43856de0b8f",
"start_timestamp": 1652358299,
"deploy_path": "/",
"last_heartbeat": 1652358589051981647,
"state_name": "Disconnected"
},
"status": {
"capacity": "1.465TiB",
"available": "1.436TiB",
"used_size": "29.6GiB",
"leader_count": 517,
"leader_weight": 1,
"leader_score": 517,
"leader_size": 48083,
"region_count": 616,
"region_weight": 1,
"region_score": 71515.07450951572,
"region_size": 60006,
"slow_score": 1,
"receiving_snap_count": 4,
"start_ts": "2022-05-12T12:24:59Z",
"last_heartbeat_ts": "2022-05-12T12:29:49.051981647Z",
"uptime": "4m50.051981647s"
}
}
]
}

What version of PD are you using (pd-server -V)?

/ # /pd-server -V
Release Version: v6.1.0-nightly
Edition: Community
Git Commit Hash: 6554d63
Git Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time: 2022-05-11 10:56:02

[mayjiang0203]

/severity Major
/assign @rleungx
/found automation

[ti-chi-bot]

@mayjiang0203: There are no type labels on this issue. Please add an appropriate label by using the following command:

• /type <type-name>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nolouch]

/check-issue-triage-complete

[nolouch]

/check-issue-triage-complete