Self-hosted Kubernetes clusters are vulnerable to the following catastrophic failure scenarios:
- Loss of all api-servers
- Loss of all schedulers
- Loss of all controller-managers
- Loss of all self-hosted etcd nodes
To minimize the likelihood of any of the these scenarios, production self-hosted clusters should always run in a high-availability configuration (TODO: add documentation for running high-availability self-hosted clusters).
Nevertheless, in the event of a control plane loss the bootkube project
provides limited disaster avoidance and recovery support through the
pod-checkpointer program and the
bootkube recover subcommand.
The Pod Checkpointer is a program that ensures that existing local pod state can be recovered in the absence of an api-server.
This is accomplished by managing "checkpoints" of local pod state as static pod manifests:
- When the checkpointer sees that a "parent pod" (a pod which should be checkpointed), is successfully running, the checkpointer will save a local copy of the manifest.
- If the parent pod is detected as no longer running, the checkpointer will "activate" the checkpoint manifest. It will allow the checkpoint to continue running until the parent-pod is restarted on the local node, or it is able to contact an api-server to determine that the parent pod is no longer scheduled to this node.
A Pod Checkpointer DaemonSet is deployed by default when using
bootkube render to create cluster manifests. Using the Pod Checkpointer is highly
recommended for all self-hosted clusters to ensure node reboot resiliency.
For more information, see the Pod Checkpointer README.
In the event of partial or total self-hosted control plane loss,
bootkube recover may be able to assist in re-bootstrapping the self-hosted control
bootkube recover subcommand does not recover a cluster directly. The
recovery is a two step process:
bootkube recover then
bootkube start. The
recovery command extracts the control plane configuration from an available
source and renders manifests to the local filesystem. These resulting manifests
can be passed to
There are two available sources to choose from in
recover: etcd or API server.
What does bootkube recover do?
bootkube recoverattempts to read the configuration from an existing backend etcd or
API server. On success,
bootkube recover writes manifests for a modified
bootstrap control plane to a directory. The second phase of the recover can be
initiated by an administrator by running
bootkube start on these manifests.
bootkube recover modifies bootstrap pod specs in the following ways:
- Ensure the pod runs as root
- Ensure the container runs as root
- Change Secret volume mounts to point to file mounts
- Change ConfigMaps volume mounts to point to file mounts
- Ensures the commandline of the containers contains --kubeconfig=/kubeconfig/kubeconfig
- Add a mount for the kubeconfig
- Bootstrap Daemonsets
- Bootstrap Deployments
- Required ConfigMaps
- Required Secrets
bootkube start to recover the cluster,
bootkube start will
automatically tear down the recovery control plane.
bootkube recover usage
For best results always use the most recently tagged Bootkube release when using
regardless of which release was used to create the cluster. To see the available releases,
checkout the tagged binary releases on GitHub
or the tagged Docker images on Quay.io.
To see available options, run:
bootkube recover --help
To recover a cluster, first invoke
bootkube recover with flags corresponding
to the current state of the cluster (supported states listed below). Then,
bootkube start to reboot the cluster. For example:
scp bootkube user@master-node: ssh user@master-node ./bootkube recover --recovery-dir=recovered [scenario-specific options] sudo ./bootkube start --asset-dir=recovered
bootkube start invocation will print the following warning message:
WARNING: recovered/manifests does not exist, not creating any self-hosted assets.
This message can be safely ignored. It is printed because recovery does not attempt to recreate self-hosted assets; it only runs a temporary control plane to allow the self-hosted control plane to recover itself.
For complete recovery examples see the hack/multi-node/bootkube-test-recovery and
If an api-server is still running
If an api-server is still running but other control plane components are down, preventing cluster functionality (i.e. the scheduler pods are all down), the control plane can be extracted directly from the api-server:
bootkube recover --recovery-dir=recovered --kubeconfig=/etc/kubernetes/kubeconfig
If an external etcd cluster is still running
If using an external etcd cluster, the control plane can be extracted directly from etcd:
bootkube recover --recovery-dir=recovered --etcd-servers=http://127.0.0.1:2379 --kubeconfig=/etc/kubernetes/kubeconfig
If an etcd backup is available (non-self-hosted etcd)
First, recover the external etcd cluster from the backup. Then use the method described in the previous section to recover the control plane manifests.