systemd node spec proposal #17688

derekwaynecarr · 2015-11-23T22:50:46Z

The following outlines changes that I want to make to the kubelet in order to better integrate with systemd systems, and to better isolate containers in their own cgroup based on the qos tier.

I think this is a precursor to getting more intelligent low compute resource eviction.

/cc @smarterclayton @ncdc @pmorie @dchen1107 @vishh @bgrant0607

This change is

dchen1107 · 2015-11-23T23:21:15Z

cc/ @jonboulle to make sure this is align with rkt

yifan-gu · 2015-11-23T23:29:29Z

cc @yifan-gu @alban

alban · 2015-11-24T13:24:03Z

docs/proposals/systemd-nodespec.md

+* ```service``` - a file that describes how to manage a service or application on the server; how to start/stop a service; when it should
+be started; under what circumstances it should be restarted; and any resource controls that should be applied to the service.
+* ```slice``` - a file that describes the inner-leaves of the cgroup hierarchy.  the name aligns with its placement in the hierarchy tree;
+for example, ```kubelet-besteffort.slice``` denotes a cgroup node ```/sys/fs/cgroup/<controller>/kubelet.slice/besteffort.slice```.  A


It actually denotes a cgroup node /sys/fs/cgroup/<controller>/kubelet.slice/kubelet-besteffort.slice. The kubelet part is repeated in the cgroup path.

Example: foo-bar.slice is a slice that is located within foo.slice, which in turn is located in the root slice -.slice.

http://www.freedesktop.org/software/systemd/man/systemd.slice.html

@alban - thanks for the typo catch! will update.

vishh · 2015-11-24T14:57:24Z

I'd prefer kubelet having complete access to croups rather than be bound to systemd cgroups APIs, which is not complete yet.

vishh · 2015-11-24T15:00:09Z

I'd suggest splitting this proposal into a separate bootstrapping/setup phase and a runtime phase.
The bootstrapping suggestions seems fine to me, but the runtime cgroups setup has some issues.
Let's get the first part of of the way, since the runtime is anyways still controlled by docker as of now.

jonboulle · 2015-11-24T18:43:18Z

I'd prefer kubelet having complete access to croups rather than be bound to systemd cgroups APIs, which is not complete yet.

This is a pretty significant decision either way. On the rkt side we'd definitely have a preference for sticking to the systemd cgroups API for now since it's a lot simpler and provides a much cleaner abstraction/integration with the way rkt pods are structured. I think as was discussed in the systemd integration meeting the other week, now that unified hierarchy is landing we're at a point where we could start to push on upstream to add the different things we need.

But if it's a dealbreaker we can make something work the other way...

derekwaynecarr · 2015-11-24T19:33:49Z

@vishh - I know we discussed setting up memory soft limits (which is in systemd, was not in docker last I looked, but looking at master it looks like some work went into adding --reservation field; i need to verify the release that is actually in). Are there specific controllers or properties not yet exposed that you want to exploit in the near term?

Either way, I agree splitting the proposal into the phases proposed.

Thanks!

pmorie · 2015-11-25T18:36:35Z

docs/proposals/systemd-nodespec.md

+
+**Status**: Proposed
+
+*This document presents a specification for how the ```kubelet``` interfaces with ```systemd``` with a focus on quality of service concerns.*


nit: triple backticks unnecessary, single is fine.

pmorie · 2015-11-25T18:40:47Z

Nit: wrapping lines at 100 chars helps make proposals easier to comment on.

vishh · 2015-11-30T20:47:45Z

@derekwaynecarr: Acknowledged.

derekwaynecarr · 2015-12-04T23:53:03Z

@vishh - I hope this clarifies what I am looking to enable.

@mrunalp - your review is appreciated as well.

Anyone else please take a look and provide input.

vishh · 2015-12-05T00:04:38Z

docs/proposals/kubelet-systemd.md

+
+## kubelet cgroups per quality of service (qos) tier
+
+It is desired to parent containers in a cgroup based on their qos tier to facilitate local node accounting scenarios.


@derekwaynecarr: Kubelet doesn't do any qos specific cgroup management as of now. Until the overall plan for cgroup management is finalized, can we exclude this section from this proposal?
Let's get the node initial bootstrapping finalized while the cgroups part gets finalized.
WDYT?

That's fine , hope idea was clear for future discussion topics. I just wanted to relate with cgroup-root that we have today.

derekwaynecarr · 2016-03-16T19:49:43Z

I am going to take another pass at this to update based on the state of Kubernetes 1.2.

Aspects of this document are stale.

derekwaynecarr · 2016-04-05T21:12:42Z

add a point on volume storage driver daemons:
#23491 (comment)

derekwaynecarr · 2016-05-19T20:46:58Z

one more item to note:

when systemd manages docker, and resource accounting is off for that unit, the container runtime stats defaults to the cpu cgroup container, which will be /, which means its the same as node level stats. We need to require that the unit file that manages docker has the following:

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target

[Service]
CPUAccounting=true
MemoryAccounting=true

Otherwise, runtime accounting will be wrong.

dchen1107 · 2016-05-19T21:15:43Z

cc/ @adityakali @andyzheng0831 this is the proposal I mentioned to you earlier related to NodeAllocatable configuration for GCI image.

andyzheng0831 · 2016-05-19T21:46:06Z

@dchen1107 not sure if this proposal covers cAdvisor. So far I feel cAdvisor does not work well with systemd in some cases.

dchen1107 · 2016-05-19T22:21:18Z

@andyzheng0831 The proposal doesn't cover cAdvisor since it is part of kubelet today. This one is trying to standardize systemd node configuration, especially on resource management side.

dchen1107 · 2016-05-19T22:28:28Z

docs/proposals/kubelet-systemd.md

+
+### Docker runtime support for --cgroup-parent
+
+Docker versions <= 1.0.9 did not have proper support for `-cgroup-parent` flag on `systemd`.  This


Docker 1.7 is minimal version Kubelet support since 1.1 release. Kubelet will mark the node not_ready today if docker version is below that minimal version.

dchen1107 · 2016-05-19T22:30:47Z

LGTM

vishh · 2016-05-19T22:56:54Z

@andyzheng0831 @dchen1107 It would help enumerate cAdvisor issues to see if any of them falls within the scope of this PR. WDYT?

k8s-bot · 2016-05-19T23:20:47Z

GCE e2e build/test passed for commit eee9a58.

k8s-github-robot · 2016-05-20T00:13:21Z

Automatic merge from submit-queue

derekwaynecarr · 2016-05-20T00:44:12Z

I am happy to look at cAdvisor issues if enumerated. I had added some stuff this release to ignore .mount cgroups. Happy to do others if there is a known list.

@smarterclayton

Automatic merge from submit-queue systemd node spec proposal The following outlines changes that I want to make to the ```kubelet``` in order to better integrate with ```systemd``` systems, and to better isolate containers in their own ```cgroup``` based on the qos tier. I think this is a precursor to getting more intelligent low compute resource eviction. /cc @smarterclayton @ncdc @pmorie @dchen1107 @vishh @bgrant0607  --- This change is [<img src="http://reviewable.k8s.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](http://reviewable.k8s.io/reviews/kubernetes/kubernetes/17688)

googlebot added the cla: yes label Nov 23, 2015

k8s-github-robot assigned bgrant0607 Nov 23, 2015

k8s-github-robot added kind/design Categorizes issue or PR as related to design. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 23, 2015

alban reviewed Nov 24, 2015
View reviewed changes

pmorie reviewed Nov 25, 2015
View reviewed changes

derekwaynecarr mentioned this pull request Nov 30, 2015

Define kubernetes node spec for systemd hosts #17556

Closed

bgrant0607 assigned dchen1107 and unassigned bgrant0607 Nov 30, 2015

derekwaynecarr force-pushed the systemd_nodespec branch 2 times, most recently from a6d0d9a to 47c56b5 Compare December 4, 2015 23:46

derekwaynecarr changed the title ~~WIP: systemd node spec proposal~~ systemd node spec proposal Dec 4, 2015

vishh reviewed Dec 5, 2015
View reviewed changes

derekwaynecarr mentioned this pull request Mar 30, 2016

WIP: Initializers proposal #17305

Closed

k8s-github-robot added the release-note-label-needed label Mar 31, 2016

derekwaynecarr added this to the v1.3 milestone May 5, 2016

k8s-github-robot assigned smarterclayton and unassigned dchen1107 May 6, 2016

derekwaynecarr assigned dchen1107 and unassigned smarterclayton May 12, 2016

node spec for systemd environments

eee9a58

derekwaynecarr force-pushed the systemd_nodespec branch from 931cb0d to eee9a58 Compare May 12, 2016 21:32

derekwaynecarr mentioned this pull request May 19, 2016

Fix detection of docker cgroup on RHEL #25907

Merged

dchen1107 reviewed May 19, 2016
View reviewed changes

dchen1107 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 19, 2016

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 19, 2016

dchen1107 added release-note Denotes a PR that will be considered when it comes time to generate release notes. lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed release-note-label-needed labels May 19, 2016

k8s-github-robot merged commit 53b5b41 into kubernetes:master May 20, 2016

sdodson mentioned this pull request May 23, 2016

[RPMS] Enable CPU and Memory accounting during node installation openshift/origin#8989

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

systemd node spec proposal #17688

systemd node spec proposal #17688

derekwaynecarr commented Nov 23, 2015 •

edited by mikedanese

Loading

dchen1107 commented Nov 23, 2015

yifan-gu commented Nov 23, 2015

alban Nov 24, 2015

iaguis Nov 24, 2015

derekwaynecarr Nov 24, 2015

vishh commented Nov 24, 2015

vishh commented Nov 24, 2015

jonboulle commented Nov 24, 2015

derekwaynecarr commented Nov 24, 2015

pmorie Nov 25, 2015

pmorie commented Nov 25, 2015

vishh commented Nov 30, 2015

derekwaynecarr commented Dec 4, 2015

vishh Dec 5, 2015

derekwaynecarr Dec 5, 2015

derekwaynecarr commented Mar 16, 2016

derekwaynecarr commented Apr 5, 2016

derekwaynecarr commented May 19, 2016

dchen1107 commented May 19, 2016

andyzheng0831 commented May 19, 2016

dchen1107 commented May 19, 2016

dchen1107 May 19, 2016

dchen1107 commented May 19, 2016

vishh commented May 19, 2016

k8s-bot commented May 19, 2016

k8s-github-robot commented May 20, 2016

derekwaynecarr commented May 20, 2016


		Status: Proposed

		This document presents a specification for how the ```kubelet``` interfaces with ```systemd``` with a focus on quality of service concerns.


		## kubelet cgroups per quality of service (qos) tier

		It is desired to parent containers in a cgroup based on their qos tier to facilitate local node accounting scenarios.


		### Docker runtime support for --cgroup-parent

		Docker versions <= 1.0.9 did not have proper support for `-cgroup-parent` flag on `systemd`. This

systemd node spec proposal #17688

systemd node spec proposal #17688

Conversation

derekwaynecarr commented Nov 23, 2015 • edited by mikedanese Loading

dchen1107 commented Nov 23, 2015

yifan-gu commented Nov 23, 2015

alban Nov 24, 2015

Choose a reason for hiding this comment

iaguis Nov 24, 2015

Choose a reason for hiding this comment

derekwaynecarr Nov 24, 2015

Choose a reason for hiding this comment

vishh commented Nov 24, 2015

vishh commented Nov 24, 2015

jonboulle commented Nov 24, 2015

derekwaynecarr commented Nov 24, 2015

pmorie Nov 25, 2015

Choose a reason for hiding this comment

pmorie commented Nov 25, 2015

vishh commented Nov 30, 2015

derekwaynecarr commented Dec 4, 2015

vishh Dec 5, 2015

Choose a reason for hiding this comment

derekwaynecarr Dec 5, 2015

Choose a reason for hiding this comment

derekwaynecarr commented Mar 16, 2016

derekwaynecarr commented Apr 5, 2016

derekwaynecarr commented May 19, 2016

dchen1107 commented May 19, 2016

andyzheng0831 commented May 19, 2016

dchen1107 commented May 19, 2016

dchen1107 May 19, 2016

Choose a reason for hiding this comment

dchen1107 commented May 19, 2016

vishh commented May 19, 2016

k8s-bot commented May 19, 2016

k8s-github-robot commented May 20, 2016

derekwaynecarr commented May 20, 2016

derekwaynecarr commented Nov 23, 2015 •

edited by mikedanese

Loading