Down-peer-region can't recover when enable placement-rule policy

[Smityz]

Bug Report

What version of TiKV are you using?

v6.5.3

Steps to reproduce

1. set placement-rule:

[
  {
    "group_id": "pd",
    "id": "1",
    "start_key": "",
    "end_key": "",
    "role": "voter",
    "is_witness": false,
    "count": 2,
    "label_constraints": [
      {
        "key": "disk_type",
        "op": "in",
        "values": [
          "ssd"
        ]
      }
    ],
    "location_labels": [
      "host"
    ],
    "isolation_level": "host",
    "create_timestamp": 1706696121
  },
  {
    "group_id": "pd",
    "id": "2",
    "start_key": "",
    "end_key": "",
    "role": "follower",
    "is_witness": false,
    "count": 1,
    "label_constraints": [
      {
        "key": "disk_type",
        "op": "in",
        "values": [
          "mix"
        ]
      }
    ],
    "location_labels": [
      "host"
    ],
    "isolation_level": "host",
    "create_timestamp": 1706696121
  }
]

2. Start 8 tikv nodes, where 3 nodes are labeled disk_type=mix, and the other 5 nodes are labeled disk_type=ssd. Then load data and observe that there is no leader on the mix node, which is expected.
   
3. force scale-in one mix node, find there are lots of down-peer not recover
   
4. disable placement-rule, then down-peer starts to recover

Is this caused by my wrong placement policy or a bug of placement-rule?

[Smityz]

Once I delete the isolation_level rule, peers can recover.
The logic behind the placement-rule is quite complex. Can someone help me debug it together?

[rleungx]

Maybe you can take a look at https://docs.pingcap.com/tidb/stable/schedule-replicas-by-topology-labels#pd-schedules-based-on-topology-label to find how isolation_level works.

[Smityz]

I have familiarized myself with the rules of isolation, but I am not certain if I fully understand them.
In my opinion, under this rule, as long as there is at least one follower node whose host is different from the leader node's host, these down-peers can be recovered.

[rleungx]

Can you also paste your store labels?

[Smityz]

Can you also paste your store labels?

In our cluster, 40+ nodes were mix and 100+ nodes were SSD 'and can be queried by pd-ctl store` command.

   {
      "store": {
        "id": 801704197,
        "address": "x",
        "labels": [
          {
            "key": "host",
            "value": "x"
          },
          {
            "key": "disk_type",
            "value": "mix"
          }
        ],
        "version": "6.5.3",
        "peer_address": "x",
        "status_address": "x,
        "git_hash": "fd5f88a7fdda1bf70dcb0d239f60137110c54d46",
        "start_timestamp": 1696908201,
        "deploy_path": "x",
        "last_heartbeat": 1710229078287679784,
        "state_name": "Up"
      },
     "store": {
        "id": 7534286,
        "address": "x",
        "labels": [
          {
            "key": "host",
            "value": "x"
          },
          {
            "key": "disk_type",
            "value": "ssd"
          }
        ],
        "version": "6.5.3",
        "peer_address": "x",
        "status_address": "x",
        "git_hash": "fd5f88a7fdda1bf70dcb0d239f60137110c54d46",
        "start_timestamp": 1696908176,
        "deploy_path": "x",
        "last_heartbeat": 1710229083389078969,
        "state_name": "Up"
      },

And this bug can be easily reproduced.