Feature request: dynamic cluster addon resizer

[a-robinson]

People create clusters at a wide range of sizes. People then resize clusters to a wide range of sizes. Our cluster addons are configured statically, and do not respond to such changes in size. This causes problems in some cases:

1. The default addons don't all fit on sufficiently small clusters (e.g. create a cluster with a single f1-micro on GCE, prepare to feel the pain)
2. The default addons waste a large proportion of resources on small clusters -- does my two node cluster 300MiB for heapster? Will it ever need more than one DNS pod?
3. The default addons don't scale properly to large clusters - heapster with 300MiB of memory isn't going to cut it on even some medium sized clusters, let alone clusters with a hundred nodes (example). More than one DNS pod will be useful for availability.

Some really simple control logic should be able to make the situation better by listing the nodes in the cluster and updating the addon RCs using a few basic rules. It could be part of the node controller, or be a very small container that runs on the master or even in the user's cluster (but it better be very small to justify its benefits).

@roberthbailey @zmerlynn

[davidopp]

It sounds like there are two problems
(1) We do not take cluster size into account when setting vertical and horizontal dimensions of the cluster addons at cluster creation time
(2) Even if we did (1), people might resize their cluster later, requiring us to resize the addons

I think (1) is more important than (2) right now, but (2) may become more important if people start using cluster autoscaling. (1) seems like something we should work on soon, while (2) could be deferred unless people are already doing a lot of manual cluster resizing.

@dchen1107 @yujuhong

[a-robinson]

Yes, that's true.

Although on the other hand, solving (2) also solves (1), and isn't much more difficult if we're already coming with logic for determining at which size cutoffs we do different things.

[dchen1107]

Yes, I brought this up to @roberthbailey and @zmerlynn before for 1.0 release, and pushed #7046 to 1.0 milestone. But there is no dynamic resizing. We need to address this issue soon to meeting our scalability goal.

[davidopp]

@dchen1107 #7046 looks more like it's about upgrading binary version of an addon, not changing number of replicas or resource request. But I agree they are somewhat related.

@a-robinson It seems like solving (1) simpler, i.e. it could be done statically in the setup scripts, whereas (2) requires a continuously-running control loop. (Unless I am misunderstanding.)

[yujuhong]

Even if we do scale the addons based on the cluster size, there'd be cases where they still don't fit in the cluster. We need a minimum requirement spec for a cluster.

[a-robinson]

@davidopp your understanding matches mine, I think we just have different expectations around how much work is involved in putting the logic into a control loop.

@yujuhong I checked into this for GKE, so in case it helps with a more general spec, this is the current state of our addon / system component resource usage:
CPU:
DNS: 100m + 100m + 100m + 10m = 310m
UI: 100m
Heapster: 100m
Fluentd: 100m per node
Kubelet: ??? per node
Docker: ??? per node
Memory:
DNS: 50Mi + 50Mi + 50Mi + 20Mi = 170Mi
UI: 50Mi
Heapster: 300Mi
Fluentd: 200Mi per node
Kubelet: 70Mi per node (but not actually limited by a cgroup AFAIK)
Docker: 30Mi per node (but not actually limited by a cgroup AFAIK)

On GCE, a single g1-small can handle all of this with a not unreasonably tiny amount of room to spare, but f1-micros don't really work unless you have at least three of them. All larger instance types are fine.

[piosz]

cc @jszczepkowski @mwielgus @fgrzadkowski

[derekwaynecarr]

cc @kubernetes/rh-cluster-infra @derekwaynecarr

[a-robinson]

We don't necessarily need dynamic sizing, but I'm bumping this to p1 to at least have smarter sizing on startup since I've now personally had to help multiple customers having issues with this