Kubelet: Create a top-level container for pods

[vmarmol]

Thanks to @vishh's awesome work, moby/moby#11428 went in with cgroup_parent support! With Docker 1.6 we'll be able to make top-level containers for pods.

[dchen1107]

Great! I assume you are talking about one top-level kubernetes container as parent for all kubernetes containers to protect our daemons, not per pod container here.

[vishh]

A top level cgroup for placing a bounding bucket across all pods mill most
definitely help with node stability.

On Thu, Mar 19, 2015 at 3:01 PM, Dawn Chen notifications@github.com wrote:

Great! I assume you are talking about one top-level kubernetes container
as parent for all kubernetes containers to protect our daemons, not per pod
container here.

—
Reply to this email directly or view it on GitHub
#5671 (comment)
.

[vmarmol]

I was referring to the per-pod containers :) I think if we have that we won't need the container for all kubernetes containers.

[dchen1107]

Then this is not for v1.0. In v1.0 there is no resource limit on pod, it doesn't help on node stability at all.

[vishh]

If users don't set any limits on pods, then per-pod cgroup doesn't provide much benefit. Reserving some resource for the system daemons will benefit node stability in the short-term though. The bounding box will help us implement QOS tiers in the long run.

[vmarmol]

The bounding box is gonna be a nightmare to migrate off of when we do do per pod limits. We can set aside space today for daemons and allocate the other space for the containers.

I don't disagree that this is not v1, but I don't think we need bounding-box for v1 either.

[bgrant0607]

@vmarmol Would we need to eliminate the bounding box when implementing per-pod limits?

I'd also like a bounding box to protect pods/containers with limits from those without, by putting the latter into a box.

[bgrant0607]

Ref #147, #168.

[vishh]

I discussed with @vmarmol offline and explained the rationale behind nested
bounding boxes.

On Thu, Mar 19, 2015 at 4:48 PM, Brian Grant notifications@github.com
wrote:

Ref #147 #147,
#168 #168.

—
Reply to this email directly or view it on GitHub
#5671 (comment)
.

[vmarmol]

For representing QoS tiers, the bounding boxes make sense. Having a top-level container for all containers doesn't buy is much in terms of protecting the node and only partially aligns to our longer term strategies. I just want us to be careful rolling out a hierarchy plan and having that change in the medium term.

I also spoke with @dchen1107 and we agreed that short term this doesn't buy us much. The model with limit and non-limit (batch) containers makes sense as our first step (the model you mentioned). As far as we know, that is not a v1 work item and there are a few things missing from the scheduler to enable it.

@bgrant0607 no we shouldn't need to remove the bounding boxes for pod-level limits. I think whether those are container or pod-level won't affect the overall plans.

[ConnorDoyle]

Here's one more use case to consider. When Kubernetes is run as an Apache Mesos framework, we're currently spinning up at most one Kubelet per host in the same process as a custom Mesos executor (framework component responsible for interpreting and executing tasks). Mesos already manages a container that encloses the executor process, and that container's resource limits grow and shrink as tasks (pods) come and go. From our perspective, it would be great if we could pass the executor's cgroup to the Kubelet to use as its parent. That way the Kubelet could still manage a hierarchy of containers, and Mesos could provide accurate global resource accounting and isolation. Could this (or something like it) be worked into the design?

[jdef]

xref mesosphere/kubernetes-mesos#68

[vishh]

@ConnorDoyle: IIUC you want the kubelet to be the parent of all the docker containers it runs. As of now we were thinking of running all system daemons, which includes the kubelet, in a separate hierarchy from that of the docker containers. I

[jdef]

@vishh @ConnorDoyle I think what would work really well for the kubernetes-mesos integration is to be able to specify a parent cgroup in the kubelet's configuration, so that the containers launched by kubelet are parented under that cgroup. The current integration blends the kubelet and mesos executor implementations into a single executable: the mesos executor actually configures the Kubelet instance directly:

• https://github.com/mesosphere/kubernetes-mesos/blob/master/pkg/executor/service/service.go#L244

It would be great if we could pass in some sort of parentCgroup parameter there.

[vmarmol]

@jdef being able to specify where Kubelet makes top-level containers seems perfectly reasonable (and defaulting to root "/"). That may be the same as that Kubelet or another cgroup. That shouldn't be a problem.

[ConnorDoyle]

@jdef, yes that's what I had in mind but your explanation is clearer.
@vishh, that would be OK as long as everything (k8s daemons, and the top-level container for the pod containers) could be in the executor's cgroup hierarchy.

[vishh]

@jdef @ConnorDoyle: SGTM

[jdef]

we've started implementing something along these lines. the currently exposed flags of the kubelet aren't quite sufficient since all kubelet-configured container paths, for example DockerDaemonContainer, aren't exposed. we've hacked around this for now, but it would be nice if future changes to the kubelet kept in mind that the kubelet may not be the only service on the host and that other management/provisioning tooling may want the ultimate say re: host-level configuration changes.

[bgrant0607]

cc @derekwaynecarr

[vishh]

cc @dubstack Your pending proposal on pod level cgroups is tied to this upstream issue.

[goltermann]

@vishh in plan for 1.4, or punt to 1.5?

[vishh]

@goltermann Yes. This feature has been punted to v1.5

[dims]

looks like we have to punt to 1.6 :)

[vishh]

This feature is completed as of v1.6. cc @derekwaynecarr