PD automatic failover

[weekface]

When a PD peer is not health for a while (this can be obtained from PD Cluster's /pd/health API), operator should:

1. mark this peer as failure
2. invoke deleteMember api to delete this member from the pd cluster
3. increase the replicas to add a new PD peer
4. try to delete the PVC and Pod of this PD peer over and over, let StatefulSet create the new PD peer with the same ordinal, but not use the tombstone PV
5. decrease the replicas when all members are ready again

part of #47

[weekface]

I am working hard on the feature. I encountered an issue during test verification:

PD's StatefulSet podManagementPolicy is OrderedReady, so if the 0 ordinal Pod was not ready, the StatefulSet can't be scaled out, which means can't failover.

I will change the pd podManagementPolicy to Parallel before the pd failover. Which will be another pr.

[weekface]

Another issue: when failover with current pd start script, the ordinal 0 pd will start as a new pd cluster, because it will start with initial-cluster option and new empty data dir when the ordinal is 0.

Solution:

Change the pd podManagementPolicy to Parallel

Add an annotation key bootstrapping to PD's ordinal 0 Pod to indicate that the cluster is bootstrapping, and a replicas annotation to indicate the replicas of current pd cluster.

• if there are no PVCs that ordinal is greater than 0, set bootstrapping to true
• else set bootstrapping to false

Modify the pd start script

• if the ordinal is greater than 0, verify the cluster is working now before starting when cluster bootstrapping
• if the ordinal is 0 and bootstrapping is true, use initial-cluster option to start this member, else use join option to start this member instead.

@tennix @onlymellb @xiaojingchen PTAL