HA endpoints for K8s
The following components require a highly available endpoints:
- etcd cluster,
- kube-apiserver service instances.
The latter relies on a 3rd side reverse proxies, like Nginx or HAProxy, to achieve the same goal.
etcd_access_endpoint fact provides an access pattern for clients. And the
etcd_multiaccess (defaults to
True) group var controls that behavior.
It makes deployed components to access the etcd cluster members
http://ip1:2379, http://ip2:2379,.... This mode assumes the clients
do a loadbalancing and handle HA for connections.
K8s components require a loadbalancer to access the apiservers via a reverse
proxy. Kubespray includes support for an nginx-based proxy that resides on each
non-master Kubernetes node. This is referred to as localhost loadbalancing. It
is less efficient than a dedicated load balancer because it creates extra
health checks on the Kubernetes apiserver, but is more practical for scenarios
where an external LB or virtual IP management is inconvenient. This option is
configured by the variable
loadbalancer_apiserver_localhost (defaults to
False, if there is an external
You may also define the port the local internal loadbalancer uses by changing,
nginx_kube_apiserver_port. This defaults to the value of
kube_apiserver_port. It is also important to note that Kubespray will only
configure kubelet and kube-proxy on non-master nodes to use the local internal
If you choose to NOT use the local internal loadbalancer, you will need to
configure your own loadbalancer to achieve HA. Note that deploying a
loadbalancer is up to a user and is not covered by ansible roles in Kubespray.
By default, it only configures a non-HA endpoint, which points to the
access_ip or IP address of the first server node in the
It can also configure clients to use endpoints for a given loadbalancer type.
The following diagram shows how traffic to the apiserver is directed.
Note: Kubernetes master nodes still use insecure localhost access because there are bugs in Kubernetes <1.5.0 in using TLS auth on master role services. This makes backends receiving unencrypted traffic and may be a security issue when interconnecting different nodes, or maybe not, if those belong to the isolated management network without external access.
A user may opt to use an external loadbalancer (LB) instead. An external LB
provides access for external clients, while the internal LB accepts client
connections only to the localhost.
Given a frontend
VIP address and
IP1, IP2 addresses of backends, here is
an example configuration for a HAProxy service acting as an external LB:
listen kubernetes-apiserver-https bind <VIP>:8383 option ssl-hello-chk mode tcp timeout client 3h timeout server 3h server master1 <IP1>:6443 server master2 <IP2>:6443 balance roundrobin
Note: That's an example config managed elsewhere outside of Kubespray.
And the corresponding example global vars for such a "cluster-aware" external LB with the cluster API access modes configured in Kubespray:
apiserver_loadbalancer_domain_name: "my-apiserver-lb.example.com" loadbalancer_apiserver: address: <VIP> port: 8383
Note: The default kubernetes apiserver configuration binds to all interfaces,
so you will need to use a different port for the vip from that the API is
listening on, or set the
kube_apiserver_bind_address so that the API only
listens on a specific interface (to avoid conflict with haproxy binding the
port on the VIP adddress)
This domain name, or default "lb-apiserver.kubernetes.local", will be inserted
/etc/hosts file of all servers in the
k8s-cluster group and wired
into the generated self-signed TLS/SSL certificates as well. Note that
the HAProxy service should as well be HA and requires a VIP management, which
is out of scope of this doc.
There is a special case for an internal and an externally configured (not with Kubespray) LB used simultaneously. Keep in mind that the cluster is not aware of such an external LB and you need no to specify any configuration variables for it.
Note: TLS/SSL termination for externally accessed API endpoints' will not
be covered by Kubespray for that case. Make sure your external LB provides it.
Alternatively you may specify an externally load balanced VIPs in the
supplementary_addresses_in_ssl_keys list. Then, kubespray will add them into
the generated cluster certifactes as well.
Aside of that specific case, the
loadbalancer_apiserver considered mutually
Access API endpoints are evaluated automagically, as the following:
|Local LB (default)||https://bip:sp||https://lc:nsp||https://m.aip:sp|
|Local LB + Unmanaged here LB||https://bip:sp||https://lc:nsp||https://ext|
|External LB, no internal||https://bip:sp||https://lb:lp||https://lb:lp|
|No ext/int LB||https://bip:sp||https://m.aip:sp||https://m.aip:sp|
m- the first node in the
lb- LB FQDN,
ext- Externally load balanced VIP:port and FQDN, not managed by Kubespray;
bip- a custom bind IP or localhost for the default bind IP '0.0.0.0';
nsp- nginx secure port,
nginx_kube_apiserver_port, defers to
sp- secure port,
lp- LB port,
loadbalancer_apiserver.port, defers to the secure port;
ip- the node IP, defers to the ansible IP;
access_ip, defers to the ip.
A second and a third column represent internal cluster access modes. The last column illustrates an example URI to access the cluster APIs externally. Kubespray has nothing to do with it, this is informational only.
As you can see, the masters' internal API endpoints are always
contacted via the local bind IP, which is
Note that for some cases, like healthchecks of applications deployed by
Kubespray, the masters' APIs are accessed via the insecure endpoint, which
consists of the local