Consider evict leader after offline the store

[nolouch]

Feature Request

Describe your feature request related problem

TiKV snapshot has an IO limit of 100MB/s. it's may become a bottleneck for sending snapshots. in the big cluster, we have encountered it about add learner too slow.

Describe the feature you'd like

• Consider evicting leader after offline the store, then we can get snapshot from anthoer store.

Teachability, Documentation, Adoption, Migration Strategy

• Go

[xuhui-lu]

Hey @nolouch, is this issue still open? If so, I can spend some time this week to help implement it.

[nolouch]

Thanks, @xuhui-lu, you can do it. and if you have any problem, you can ask about it in this issue.

[rleungx]

Hey @nolouch, is this issue still open? If so, I can spend some time this week to help implement it.

There is a case that needs to pay attention to. If the number of offline stores is more than the rest, evict leaders may be not a good way.

[xuhui-lu]

Hey @nolouch, is this issue still open? If so, I can spend some time this week to help implement it.

There is a case that needs to pay attention to. If the number of offline stores is more than the rest, evict leaders may be not a good way.

oh, right. If the number of offline stores is more than the rest, evict leaders will lead to continuous election failures.

[nolouch]

I found a corner case that if we deploy the cluster in cross-region. evict leader may let the snapshot task cross region，It would be worse. maybe the better solution is the follower replica tikv/raft-rs#136. Do we have any plan to support it? cc @BusyJay

[nolouch]

But the problem also exists in the store is down in the cross-region situation. the optimize can be done in the future.