Garden Cluster Disaster Recovery (a.k.a. Gardener Ring)

[vlerenc]

Story

• As operator I want my garden cluster (function-wise) to recover from a disaster, so that I can ensure business continuity.

Motivation

Availability SLO, Business continuity.

Acceptance Criteria

• Garden cluster functionality recovers from single zone loss (=partial disaster) with a Recovery Time Objective (RTO) of 5 minutes.

Implementation Proposal

Form a highly available distributed self-healing Gardener ring that watches its hosting garden clusters (forming a ring of seed/shoots) and repairs lost clusters underneath itself autonomously and automatically. This is not so much a pure disaster recovery as a mix of:

• HA Gardener
• Self-healing of rather than complete relocation to a different garden cluster (works only if the affected zone becomes available again)

Idea how this could be set up: Let's use Minikube (on an IaaS VM with IaaS LBs) to bootstrap a Gardener, create a ring of seeds (actually garden clusters) and transfer the first seed control plane into the last seed cluster, finally deploy an etcd cluster across the garden clusters, an API server as well, form a virtual distributed node-less Kubernetes cluster and transfer the Gardener control plane into that ring and shut down the Minikube cluster again. The result should be 3 seeds (actually garden clusters), one watching the other with a distributed Gardener using a distributed etcd on a distributed virtual node-less Kubernetes cluster consisting only of API servers. The whole thing becomes a self-healing Gardener ring.

Positive side effect: We can reduce our effort into Kubify and focus even more on the Gardener. Also, we can leverage the Gardener functionality for the clusters that run Gardener itself, which is an immense improvement since day-2 operations are much stronger in Gardener than they are in Kubify for obvious reasons. In addition, Gardener is more thoroughly quality-tested and production-hardened than Kubify, which in comparison runs few (but nonetheless critical garden) clusters.

Prerequisites/Requirements

• DNS/Watchdog Controller (redirecting DNS entries to active components): https://<redacted>/kubernetes/garden-ring-controller/issues/1
• Multi-Node ETCD Backup & Restore: Multi-Node ETCD Backup & Restore etcd-backup-restore#33

Resources

• Extended Shoot Cluster Control Plane Backup & Restore Design Discussion: https://github.com/amshuman-kr/gardener/blob/master/docs/proposals/shoot-control-plane-migration/README.md

Release Notes

- In case of a partial garden cluster outage or loss (partial disaster limited to one zone), Gardener can now self-heal autonomously and automatically to ensure business continuity with an RTO of 5 minutes.

Definition of Done

• Knowledge is distributed: Have you spread your knowledge in pair programming/code review?
• Unit tests are provided: Have you written automated unit tests?
• Integration tests are provided: Have you written automated integration tests?
• Minimum API exposure: If you have added/changed public API, was it really necessary/is it minimal?
• Operations guide: Have you updated the operations guide about ops-relevant changes?
• User documentation: Have you updated the READMEs/documentation about user-relevant changes?

[rfranzke]

/close as it's unlikely that we will implement this ring due to complexity concerns