Auto-balance region leader number.

[siddontang]

We can support base leader auto-balance at first, we don't consider balance region here now, and we don't consider the store machine load too, only focus leader number auto-balance.

Assume we have 3 stores (1, 2, 3) and the max peer count is 3.

Note that the balance is approximate, e,g, store 1 may have 10 leaders, 2 has 11 leaders and 3 has 9 leaders.

How to:

• Support leader transfer command.
• The leader peer in a region will report HeartBeat to pd, so pd can know how many leaders in a store.
• Pd will do auto-balance leaders regularly (the interval may be set in config file or passed by flag), or be invoked with Restful API manually later.
• Support rule, e,g, we only want leaders are all in store 1, 2, so we can't transfer leader to 3.

Problems:

• How to determine to transfer a leader from one store to another? E,g, store 1 has 100 leaders, but 2 has only 1, we may know that we should transfer some leaders from 1 to 2.
• If we know that we should do from 1 to 2, how to select the region leaders? Random is ok?

/cc @ngaut @qiuyesuifeng @disksing

[disksing]

The leader peer in a region will report HeartBeat to pd, so pd can know how many leaders in a store.

Does it mean that we have to cache all regions in memory? Will the data become too big?

[siddontang]

Yes, but don't worry, the memory can't become too huge.
Btw, we can introduce boltdb or other db to cache if we really meet the memory problem.

[disksing]

How to determine to transfer a leader from one store to another? E,g, store 1 has 100 leaders, but 2 has only 1, we may know that we should transfer some leaders from 1 to 2.
If we know that we should do from 1 to 2, how to select the region leaders? Random is ok?

Maybe we could evaluate time cost of each region on tikv and sync it to pd server.