placement: improve regoin fit (#2639)

[ti-srebot]

cherry-pick #2639 to release-4.0

---

What problem does this PR solve?

When we fit region's peers into rules, we check rules one by one, and for each one, we pick the best option (if there are multiple best options we simply pick the first one). The problem is that the current choice will affect subsequent rules, so the current optimal option is not necessarily the global optimal option.

For example, a region has 4 peers: [host=h1, host=h2, host=h3, host=h4], and it matches 2 rules: 3 voters + 1 voter on h2. The best choice is [host=h1, host=h3, host=h4] goes to the first rule and [host=h2] goes to the second rule. Without consider globally, the first rule may pick [host=h1, host=h2, host=h3] and left [host=h4] which is not suitable for the second rule.

The problem is originally brought up from #2626

What is changed and how it works?

Recursive all options to choose the global best.

Check List

Tests

• Unit test

Release note

• Fix the issue that when enabling placement rules, sometimes region replicas cannot schedule to optimal

[JmPotato]

LGTM

[ti-srebot]

@JmPotato,Thanks for your review. However, LGTM is restricted to Reviewers or higher roles.See the corresponding SIG page for more information. Related SIGs: scheduling(slack).

[nolouch]

LGTM

[codecov-commenter]

Codecov Report

Merging #2720 into release-4.0 will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           release-4.0    #2720      +/-   ##
===============================================
+ Coverage        77.45%   77.46%   +0.01%     
===============================================
  Files              206      206              
  Lines            22473    22490      +17     
===============================================
+ Hits             17406    17422      +16     
  Misses            3757     3757              
- Partials          1310     1311       +1     

Impacted Files	Coverage Δ
server/schedule/placement/fit.go	91.48% <100.00%> (+1.16%)	⬆️
server/schedule/placement/rule_manager.go	75.32% <100.00%> (ø)
pkg/etcdutil/etcdutil.go	78.26% <0.00%> (-5.80%)	⬇️
server/tso/tso.go	78.94% <0.00%> (-5.27%)	⬇️
server/grpc_service.go	60.23% <0.00%> (-0.58%)	⬇️
client/client.go	71.95% <0.00%> (-0.49%)	⬇️
server/server.go	75.78% <0.00%> (-0.45%)	⬇️
server/member/leader.go	73.15% <0.00%> (-0.39%)	⬇️
pkg/btree/btree.go	87.65% <0.00%> (+0.80%)	⬆️
pkg/mock/mockhbstream/mockhbstream.go	90.76% <0.00%> (+1.53%)	⬆️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5705eaa...3c46dcd. Read the comment docs.

[jebter]

/run-all-tests

[jebter]

/merge

[ti-srebot]

/run-all-tests