apiservers should set --advertise-address to the loadbalancer that fronts all api-servers

[aaronlevy]

If each apiserver has a different advertise address (its own host address), this means that all apiservers will fight over the same endpoint record, constantly rewriting it. This can be seen by launching a cluster with multiple masters then running kubectl get endpoints a few times (you should see the endpoint IP changing).

While this unto itself is generally okay (when hitting https://kubernetes, or the kubernetes service-ip - you will get one of the apiservers as an endpoint) - it can cause performance problems related to the kube-proxy.

Because this kube-proxy is watching for changes to endpoints, the constant re-writing of this record means that the proxy must re-write all iptables rules each time it sees a change (all rules are written as a single atomic action). If you have many services this constant iptables rule writing can become a performance hit (unfortunately, I don't have any data on specifics of how many services this can become a problem and by how much).

Unfortunately, the --advertise-address only allows an IP -- so one option would be to have an elasticIP associated with a loadbalancer.

[s-urbaniak]

As far as I know it is not possible to assign elastic IPs to load balancers in AWS, but only to single instances (/cc @alexsomesan to reassure). On the other hand ELBs do not provide single fixed IPs, so that is a dead-end too unfortunately.

Having not the possibility to set --apiserver-count (see [1]) or any other method to assign multiple IPs to the API server) I therefore suggest to simply configure the IP address of the bootstrap node as --advertise-address.

This is a pragmatic solution to cut down constant updates of iptables rules, but obviously comes with the cost that the internal API server service endpoint is not HA-aware.

I am aware that this is not the optimal solution but I'd expect a working upstream solution to allow us to set multiple IP addresses for the API server.

[1] #136

[s-urbaniak]

To add a final note to the above: as far as I remember our "recommended" way to talk to the api server is using DNS names and load balancers anyways, nevertheless again pointing out that the suggested solution is not optimal.

[sym3tri]

@s-urbaniak yeah I think you're right, there is nowhere to assign an EIP to ELBs in the AWS Console. @alexsomesan please confirm.

Looks like this will be an ongoing issue with less than ideal workarounds at best until this is resolved upstream.

@aaronlevy any suggestions? What do you think of Serg's workaround?

[aaronlevy]

We definitely cannot set all advertise addresses to the same node -- if that goes down the control plane is down (why bother with HA at all then?). The churn in apiservers is better than non-HA apiservers.

[aaronlevy]

FWIW the best numbers I've seen around proxy churn is this presentation: https://docs.google.com/presentation/d/1BaIAywY2qqeHtyGZtlyAp89JIZs59MZLKcFLxKE6LyM

Granted the normal cluster is going to be nowhere near those numbers of services - but at least gives an idea.

[s-urbaniak]

Sorry for the copy-pasta, but I didn't want this information to get lost, in case we decide against #467 and close it. As @aaronlevy points out my suggested work-around only points to the bootstrapping node, and I dislike that fact a lot too.

I did not find a way yet to get a fixed IP for AWS ELBs, here a couple of external references I stumbled upon:

• https://www.pagerduty.com/blog/load-balancers-need-static-ips/
• https://shlomoswidler.com/2009/07/elastic-in-elastic-load-balancing-elb.html
• http://stackoverflow.com/questions/13767349/aws-load-balancer-with-a-static-ip-address
• http://stackoverflow.com/questions/35313134/assigning-static-ip-address-to-aws-load-balancer

And also the official AWS doc - https://aws.amazon.com/articles/1636185810492479:

The Elastic Load Balancing service will update the Domain Name System (DNS) record of the load balancer when it scales so that the new resources have their respective IP addresses registered in DNS. The DNS record that is created includes a Time-to-Live (TTL) setting of 60 seconds, with the expectation that clients will re-lookup the DNS at least every 60 seconds. By default, Elastic Load Balancing will return multiple IP addresses when clients perform a DNS resolution, with the records being randomly ordered on each DNS resolution request. As the traffic profile changes, the controller service will scale the load balancers to handle more requests, scaling equally in all Availability Zones.

[sym3tri]

@aaronlevy there is no known viable workaround for this. Closing until we come up with one or get this fixed upstream.

[aaronlevy]

@sym3tri Might want to leave this open to track that we still need to resolve the issue. Could change title to be less prescriptive of fix and more about the underlying issue we need to resolve (churn in apiserver endpoint)

[petergardfjall]

You may want to try the lease reconciler (available from Kubernetes 1.9) to keep apiservers from fighting over the kubernetes endpoint.
kubernetes/kubeadm#546 (comment)