Control-plane without LB for API-server is a SPOF #2103

NicolasT · 2019-12-06T13:30:08Z

When deploying without creating an external load-balancer for the API server, or enabling the keepalived VIP management, we end up configuring all services who require 'external' access to the API server, including kube-controller-manager, kube-scheduler, kubelet, kube-proxy to connect to apiServer.host from the BootstrapConfiguration, which would then be the bootstrap node. As such, this becomes a SPOF for the control-plane, even when multiple control-plane nodes are up and running.

This can be resolved by either using an external LB or use the built-in keepalived functionality. However, this may not be feasible in all environments.

As such, we could consider adopting the Kubespray approach of deploying an nginx instance on every host in the cluster (using the hostNetwork), listening on 127.0.0.1:6443, which then balances/proxies to all known API server instances in the cluster. At that point, we can configure all k-c-ms, k-ss, kubelets, kube-proxys and other consumers of the API server to connect to 127.0.0.1:6443.

This does not fix the issue of API server accessibility from outside the cluster, i.e. an admin's workstation or similar.

The text was updated successfully, but these errors were encountered:

NicolasT · 2019-12-06T13:35:19Z

The benefit of using an 'internal' HA'ish solution would be that there's no need to special-case the keepalived infrastructure we have right now: if we want to provide a stable IP towards the API server for 'real extenral' access, we can use whatever HA/LB solution we come up with for #1788 when (also) applied to the control-plane.

NicolasT · 2019-12-06T18:19:53Z

Moving forward with the nginx-based approach, we could then use this same proxy for other purposes like access to the repositories, salt-master,...

Since we'll start using `nginx` as a 'normal' container, we need to distribute it as both a tar archive (to inject in the `containerd` cache on the bootstrap node), as well as in the 'registry' for clients to pull. See: #2103

When using a local(host) `nginx` to proxy to the `kube-apiserver` instances using a simple *stream* transport (i.e., TCP socket, no TLS handling by `nginx`), we need to ensure the TLS server exposes a certificate the client will accept, i.e., has `IP:127.0.0.1` in the `subjectAlternateName` field. See: #2103

This is similar to the poor-mans-HA functionality as found in the Kubespray project. See: #2103

Instead of using `BootstrapConfiguration` `apiServer.host`, which will be removed in a subsequent commit since no longer required, use the control-plane IP of the host on which the `KubeConfig` file is generated. See: #2103

…rver` Instead of relying on the `BootstrapConfiguration` `apiServer.host` value, we can instead use the proxy to `kube-apiserver` running on every host in the cluster. See: #2103

We no longer need this since we provide in-cluster HA for `kube-apiserver` access. If this is desired for out-of-cluster access, we can provide this using a `LoadBalancer` `Service` once we have the infrastructure to support this in place. This also removed the optional deployment of `keepalived`. See: #2103 See: #1788