Providing leader election mechanisms to Pilots

[munnerz]

Some databases require cluster-wide options to be set by a single node in the cluster. For example, Elasticsearch has an 'exclude shards' field that can be set via the Elasticsearch API, that tells the cluster to evict shards from the nodes listed.

Updates to this field need co-ordinating, and so having a single cluster-aware component that watches all Pilots is desired. I'm thinking we can embed this logic into the actual Pilot itself to save deploying additional components. This gives us HA by default too (up to the degree that the database itself is HA).

Rough requirements:

• Add a ConfigMap or Endpoints like type to navigator-apiserver so leader election can be performed irrespective of the k8s API server
• Import leader election code into the genericpilot
• Provide a LeaderElectedSyncFunc capability in the genericpilot, that will be called upon every update to every Pilot within the cluster the Pilot is a member of

It's then up to the actual Pilot implementation to do something sensible here. Additionally, the leader elected sync function would be 'allowed' to update the status block of any of the Pilot resources it processes.

Interested to hear your thoughts @wallrj - is this sort of thing required to handle scale down of Cassandra clusters, or say triggering backups or any other administrative action that may need to be performed?

/kind feature