Kubernetes Master High Availability Proposal

[keontang]

High Availability Kubernetes Clusters Promotion

Introduction

There have been attempts in achieving high-availability master in the community:

• High Availability: kubelet doesn't try to talk to a different API server in case of failures
• allow the restclient to contact multiple apiservers

But the feature of multiple apiserver isn't going to make it into v1.3. Moreover, the method of a load balancer in front of the multiple apiservers is still very popular for high-availability master deployment.

So in this document we present a proposal on how to setup high-availability master using haproxy as the loadbalancer, especailly in private cloud scenarios.

It enriches the previous document (http://kubernetes.io/docs/admin/high-availability/) with more details in the following two aspects:

1. All components except kubelet are dockerized. And the kubelet is responsible for maintaining a set of these static pods, including haproxy, keepalived, apiserver, controller manager, kubeproxy, flannel and etcd.
2. Loadbalancer is implemented by haproxy and keepalived. Haproxy is configured with SSL Termination,and responsible for load balancing. Keepalived is responsible for mantaining VIP in the kubernetes cluster.

Overview

We deployed our Kubernetes clusters with HA master on Ubuntu 14.04 operating system in the testing private cloud environment. Here's example of what the environment should look like by the end of the HA master deployment:

HOST        IP ADDRESS         PROCESS    STATIC POD
lb-1        192.168.205.252    kubelet    haproxy, keepalived
lb-2        192.168.205.253    kubelet    haproxy, keepalived
master-1    192.168.205.11     kubelet    apiserver, controller manager, scheduler, kubeproxy, etcd, flannel
master-2    192.168.205.12     kubelet    apiserver, controller manager, scheduler, kubeproxy, etcd, flannel
master-3    192.168.205.13     kubelet    apiserver, controller manager, scheduler, kubeproxy, etcd, flannel
node-1      192.168.205.21     kubelet    kubeproxy, flannel
node-2      192.168.205.22     kubelet    kubeproxy, flannel
node-3      192.168.205.23     kubelet    kubeproxy, flannel

It builds virtual ip 192.168.205.254 over lb-1 and lb-2 by keepalived.

Components needed to run in the privileged mode

• keepalived: need to access IP_VS kernel modules to setup VIP
• flannel: need to access vxlan, openvswitch to route packages
• kubeproxy: need to setup iptables firewall rules
• kubelet: need to allow containers to request privileged mode
• apiserver: need to allow privileged containers to access apiservers

[lavalamp]

I don't understand what this issue is about. Are you trying to add documentation? If so please send a PR to the docs repository: https://github.com/kubernetes/kubernetes.github.io/tree/release-1.3

[mikedanese]

@keontang these are great diagrams. You should write a blog post about your experience setting up an HA kubernetes cluster.

[keontang]

@mikedanese You mean blog.kubernetes.io ? I am happy to share my experience in HA Kubernetes Cluster Building. Maybe I can also add some code to k8s.io/kuberentes/examples ?

[kshafiee]

Great post @keontang !

Do you happen to have any detailed instructions of you HA implementation? I'm trying to reproduce your HA cluster, Thx

[gaddamidianil]

@keontang Can you please provide the complete documentation for master HA setup. Thanks.

[xigang]

@keontang Can you provide a detailed set up documents? HA cluster in the recent study of this.

[DanyC97]

@lavalamp i don't get it why you closed this issue, it should have been left open until someone pushed a PR to add this detailed info to the docs.

Currently there is not blog as suggested by @mikedanese and no docs updated hence is a bit difficult for anyone who find @keontang 's comment to have a full implementation etc

Would appreciate if you reconsider re-open it.

[keontang]

Wol, so many people care about ha master, i will make the doc in my spare time.

[FengyunPan]

@keontang Have you finished the blog post about your experience setting up an HA kubernetes cluster?

[KeithTt]

Any update...? 💯

[dpoison]

@keontang Any update? Can you share your doc?