Support high availability clusters

[lavalamp]

Master components need to be replicated for high availability.

[bgrant0607]

I changed the title. Sharding the kubelet communication may be useful at large scale, but unnecessary in order to improve availability. Ditto for moving kubelet communication to a separate service.

How does the apiserver recover its state after restarting? By reloading it from etcd? Presumably apiserver replicas could do the same.

The apiserver could be replicated and use etcd for master election. We could initially make just the elected master talk to kubelets. Do we propagate resource version sequence numbers all the way down to kubelet? We'd want a way to prevent wayward former masters from giving kubelets stale/wrong commands.

[lavalamp]

Currently we have no need for master election, apiserver is stateless. I also don't think we're close to having availability problems, so we might as well take our time and solve the sharding problem-- there's no rush here.

kubelet reads objects directly from etcd, apiserver doesn't send kubelet instructions. The apiserver->kubelet path is about info gathering, not command sending.

[smarterclayton]

I think the biggest gap today is that the operations queue aren't shared between masters behind a load balancer and clients assume they can fetch operations.

[lavalamp]

Good point. How would we fix that? Have a registry/operations/ in etcd that we add an entry to every time an apiserver admits a PUT or POST? Would there be an expectation that another apiserver might pick it up if it isn't eventually marked as completed?

[smarterclayton]

We could encode something unique about each startup of the apiserver (generated UUID) and put that into etcd along with the reachable host (which I don't like as much, seems fragile). Then in the status response we could return the UUID of the process and the operation identifier and do a proxy to the server corresponding to it in etcd (assumes it's reachable). That reduces the potential cross server coupling to etcd (which is hard already) and you could say "you can't load balance across server that can't reach each other or can't talk to the same etcd".

[pires]

Using etcd to map existing apiservers seems very fragile indeed. You'd need for some sort of watchdog mechanism to remove entries related to dead/inaccessible servers. And you'd rely heavily on the availability of etcd - which you already do.

Could we think about another solution, like apiservers communicating with each other (p2p, like in a Cassandra or Hazelcast cluster) and have them gossip the state of the cluster between themselves? This would work as well for partitioning/replicating cluster data.

[smarterclayton]

We never updated this, but the agreement at the Kube face to face is to nuke operations (no operation in the api server should be long running). The ip allocator is single writer at this point - it could be converted to a sharded key write / merge operation against etcd with work. The controllers and scheduler need to elect themselves by waiting on an etcd key in the short term, and in the long term compete for work and shard work on related queues. The scheduler is the first component that must be made ha if we want to run the scheduler on the cluster (we do) since it would need to schedule it's replacement. Every control loop is what I'm referring to as controller - we still have a few that are in the master code vs in controller manager.

[erictune]

ip allocator could instead be a "finalizer"?

[erictune]

finalizers ➡️ #3585

[smarterclayton]

Yeah - clients would look at Service.Status.ServiceIP, and would say "service not ready" until it had an IP.

----- Original Message -----

ip allocator could instead be a "finalizer"?

---

Reply to this email directly or view it on GitHub:
#473 (comment)

[bgrant0607]

Operations have been eliminated.

This is not in our 1.0 roadmap, but if someone is interested in helping, we'd happily guide them.

[timothysc]

Are there any known limitations with baby stepping here? For example load-balancing the api-servers?

We're finding the api-server to be the bottleneck and can be cpu-bound even on steady state...

[smarterclayton]

I have a set of changes for service ip allocation that would allow us to move that to a control loop. I think we could try everything else there.

Do you guys have profiles yet? My recent stuff was showing conversion and serialization dominating, but it's not at your workloads.

On Mar 10, 2015, at 3:22 PM, Timothy St. Clair notifications@github.com wrote:

Are there any known limitations with baby stepping here? For example load-balancing the api-servers?

We're finding the api-server to be the bottleneck and can be cpu-bound even on steady state...

—
Reply to this email directly or view it on GitHub.

[roberthbailey]

Re-setting assignees based on who is working on HA in kubeadm for 1.8.

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Removed

@lavalamp @luxas @timothysc @wojtek-t

Important: This issue was missing labels required for the v1.9 milestone for more than 3 days:

kind: Must specify exactly one of kind/bug, kind/cleanup or kind/feature.

Help

• Additional instructions
• Commands for setting labels

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[timothysc]

I'm going to move to close this, there are recipes for recipes today and this issue no longer tracks any of the details. Different repos have assorted parent-child issues tracking the state and there has been docs published to the main site as well.