New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cAdvisor in Kube 1.4 is broken on RHEL systems when running in a container - cgroup is out of order #1461

Closed
smarterclayton opened this Issue Sep 14, 2016 · 23 comments

Comments

Projects
None yet
10 participants
@smarterclayton
Contributor

smarterclayton commented Sep 14, 2016

Kubelet start fails in cAdvisor because of an notify watch failure on what is a symlink

F0914 17:16:59.291599   18951 kubelet.go:1149] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/cpuacct,cpu: no such file or directory

https://paste.fedoraproject.org/428168/38900021/

This is from the 1.4 cAdvisor (or just before?) - when mounted in a container cadvisor sees a different cgroup path, which causes cAdvisor to quick. Breaks containerized kubelet using -v /sys:/sys mounts.

@smarterclayton

This comment has been minimized.

Show comment
Hide comment
@smarterclayton

smarterclayton Sep 14, 2016

Contributor

@derekwaynecarr @ncdc @mrunalp correct me if I got any of that wrong.

Contributor

smarterclayton commented Sep 14, 2016

@derekwaynecarr @ncdc @mrunalp correct me if I got any of that wrong.

@mrunalp

This comment has been minimized.

Show comment
Hide comment
@mrunalp

mrunalp Sep 14, 2016

@smarterclayton Yeah, the system order is different from what we see in the container when we bind mount /sys over /sys. IMO, cadvisor should check which path exists and set the inotify watch accordingly.

mrunalp commented Sep 14, 2016

@smarterclayton Yeah, the system order is different from what we see in the container when we bind mount /sys over /sys. IMO, cadvisor should check which path exists and set the inotify watch accordingly.

@vishh

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh Sep 14, 2016

Contributor

IIRC, this was fixed via a libcontainer update. Libcontainer was mounting
cgroups with non-identify mappings inside the container. I'm surprised this
is showing up in k8s v1.4.0

On Wed, Sep 14, 2016 at 3:03 PM, Mrunal Patel notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton Yeah, the system
order is different from what we see in the container when we bind mount
/sys over /sys. IMO, cadvisor should check which path exists and set the
inotify watch accordingly.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKC_FHeWF4mZKCThrV401yj9dJmcyks5qqG88gaJpZM4J9SiV
.

Contributor

vishh commented Sep 14, 2016

IIRC, this was fixed via a libcontainer update. Libcontainer was mounting
cgroups with non-identify mappings inside the container. I'm surprised this
is showing up in k8s v1.4.0

On Wed, Sep 14, 2016 at 3:03 PM, Mrunal Patel notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton Yeah, the system
order is different from what we see in the container when we bind mount
/sys over /sys. IMO, cadvisor should check which path exists and set the
inotify watch accordingly.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKC_FHeWF4mZKCThrV401yj9dJmcyks5qqG88gaJpZM4J9SiV
.

@mrunalp

This comment has been minimized.

Show comment
Hide comment
@mrunalp

mrunalp Sep 14, 2016

@vishh This is what libcontainer mounts inside a container (for e.g.):

drwxr-xr-x. 2 root root  0 Sep 14 22:18 blkio
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpu -> cpu,cpuacct
drwxr-xr-x. 2 root root  0 Sep 14 22:18 cpu,cpuacct
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpuacct -> cpu,cpuacct
drwxr-xr-x. 2 root root  0 Sep 14 22:18 cpuset
drwxr-xr-x. 2 root root  0 Sep 14 22:18 devices
drwxr-xr-x. 2 root root  0 Sep 14 22:18 freezer
drwxr-xr-x. 2 root root  0 Sep 14 22:18 hugetlb
drwxr-xr-x. 2 root root  0 Sep 14 22:18 memory
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_cls -> net_cls,net_prio
drwxr-xr-x. 2 root root  0 Sep 14 22:18 net_cls,net_prio
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_prio -> net_cls,net_prio
drwxr-xr-x. 2 root root  0 Sep 14 22:18 perf_event
drwxr-xr-x. 2 root root  0 Sep 14 22:18 pids
drwxr-xr-x. 2 root root  0 Sep 14 22:18 systemd

Now, when we bind mount /sys:/sys we are overwriting that with the order on the system which happens to be the opposite on RHEL and hence the issue.
One thing that cadvisor could do is not use combined paths as symlinks are provided for each subsystem to a combined path.

Not sure how we can fix this in libcontainer.

mrunalp commented Sep 14, 2016

@vishh This is what libcontainer mounts inside a container (for e.g.):

drwxr-xr-x. 2 root root  0 Sep 14 22:18 blkio
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpu -> cpu,cpuacct
drwxr-xr-x. 2 root root  0 Sep 14 22:18 cpu,cpuacct
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpuacct -> cpu,cpuacct
drwxr-xr-x. 2 root root  0 Sep 14 22:18 cpuset
drwxr-xr-x. 2 root root  0 Sep 14 22:18 devices
drwxr-xr-x. 2 root root  0 Sep 14 22:18 freezer
drwxr-xr-x. 2 root root  0 Sep 14 22:18 hugetlb
drwxr-xr-x. 2 root root  0 Sep 14 22:18 memory
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_cls -> net_cls,net_prio
drwxr-xr-x. 2 root root  0 Sep 14 22:18 net_cls,net_prio
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_prio -> net_cls,net_prio
drwxr-xr-x. 2 root root  0 Sep 14 22:18 perf_event
drwxr-xr-x. 2 root root  0 Sep 14 22:18 pids
drwxr-xr-x. 2 root root  0 Sep 14 22:18 systemd

Now, when we bind mount /sys:/sys we are overwriting that with the order on the system which happens to be the opposite on RHEL and hence the issue.
One thing that cadvisor could do is not use combined paths as symlinks are provided for each subsystem to a combined path.

Not sure how we can fix this in libcontainer.

@vishh

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh Sep 14, 2016

Contributor

cAdvisor uses libcontainer to identify the mount points. IIRC, libcontainer
used the first set of mount paths it identifies. I don't have time to dig
into it now, but in essence, libcontainer should be able to pick the right
set of mount points.

On Wed, Sep 14, 2016 at 3:22 PM, Mrunal Patel notifications@github.com
wrote:

@vishh https://github.com/vishh This is what libcontainer mounts inside
a container (for e.g.):

drwxr-xr-x. 2 root root 0 Sep 14 22:18 blkio
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpu -> cpu,cpuacct
drwxr-xr-x. 2 root root 0 Sep 14 22:18 cpu,cpuacct
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpuacct -> cpu,cpuacct
drwxr-xr-x. 2 root root 0 Sep 14 22:18 cpuset
drwxr-xr-x. 2 root root 0 Sep 14 22:18 devices
drwxr-xr-x. 2 root root 0 Sep 14 22:18 freezer
drwxr-xr-x. 2 root root 0 Sep 14 22:18 hugetlb
drwxr-xr-x. 2 root root 0 Sep 14 22:18 memory
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_cls -> net_cls,net_prio
drwxr-xr-x. 2 root root 0 Sep 14 22:18 net_cls,net_prio
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_prio -> net_cls,net_prio
drwxr-xr-x. 2 root root 0 Sep 14 22:18 perf_event
drwxr-xr-x. 2 root root 0 Sep 14 22:18 pids
drwxr-xr-x. 2 root root 0 Sep 14 22:18 systemd

Now, when we bind mount /sys:/sys we are overwriting that with the order
on the system which happens to be the opposite on RHEL and hence the issue.
One thing that cadvisor could do is not use combined paths as symlinks are
provided for each subsystem to a combined path.

Not sure how we can fix this in libcontainer.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKCPlMtjvkaxzSG8dzuA-xbn6cfsrks5qqHOZgaJpZM4J9SiV
.

Contributor

vishh commented Sep 14, 2016

cAdvisor uses libcontainer to identify the mount points. IIRC, libcontainer
used the first set of mount paths it identifies. I don't have time to dig
into it now, but in essence, libcontainer should be able to pick the right
set of mount points.

On Wed, Sep 14, 2016 at 3:22 PM, Mrunal Patel notifications@github.com
wrote:

@vishh https://github.com/vishh This is what libcontainer mounts inside
a container (for e.g.):

drwxr-xr-x. 2 root root 0 Sep 14 22:18 blkio
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpu -> cpu,cpuacct
drwxr-xr-x. 2 root root 0 Sep 14 22:18 cpu,cpuacct
lrwxrwxrwx. 1 root root 11 Sep 14 22:18 cpuacct -> cpu,cpuacct
drwxr-xr-x. 2 root root 0 Sep 14 22:18 cpuset
drwxr-xr-x. 2 root root 0 Sep 14 22:18 devices
drwxr-xr-x. 2 root root 0 Sep 14 22:18 freezer
drwxr-xr-x. 2 root root 0 Sep 14 22:18 hugetlb
drwxr-xr-x. 2 root root 0 Sep 14 22:18 memory
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_cls -> net_cls,net_prio
drwxr-xr-x. 2 root root 0 Sep 14 22:18 net_cls,net_prio
lrwxrwxrwx. 1 root root 16 Sep 14 22:18 net_prio -> net_cls,net_prio
drwxr-xr-x. 2 root root 0 Sep 14 22:18 perf_event
drwxr-xr-x. 2 root root 0 Sep 14 22:18 pids
drwxr-xr-x. 2 root root 0 Sep 14 22:18 systemd

Now, when we bind mount /sys:/sys we are overwriting that with the order
on the system which happens to be the opposite on RHEL and hence the issue.
One thing that cadvisor could do is not use combined paths as symlinks are
provided for each subsystem to a combined path.

Not sure how we can fix this in libcontainer.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKCPlMtjvkaxzSG8dzuA-xbn6cfsrks5qqHOZgaJpZM4J9SiV
.

@timstclair

This comment has been minimized.

Show comment
Hide comment
@timstclair

timstclair Sep 14, 2016

Contributor

Is this the same as #1444?

Contributor

timstclair commented Sep 14, 2016

Is this the same as #1444?

@mrunalp

This comment has been minimized.

Show comment
Hide comment
@mrunalp

mrunalp Sep 14, 2016

@vishh Yes, libcontainer looks at the first mount points that it finds. How is cadvisor expected to be launched in a container? What host paths are expected to be mounted inside? This will help identify how to fix this.

mrunalp commented Sep 14, 2016

@vishh Yes, libcontainer looks at the first mount points that it finds. How is cadvisor expected to be launched in a container? What host paths are expected to be mounted inside? This will help identify how to fix this.

@vishh

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh Sep 14, 2016

Contributor

It expects all the cgroups mounts from the host to be mounted. It uses
libcontainer to detect the cgroup mounts.

On Wed, Sep 14, 2016 at 4:10 PM, Mrunal Patel notifications@github.com
wrote:

@vishh https://github.com/vishh Yes, libcontainer looks at the first
mount points that it finds. How is cadvisor expected to be launched in a
container? What host paths are expected to be mounted inside? This will
help identify how to fix this.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKHHJ0rWbzBkuwzXDHd5t49NN_Kxuks5qqH7RgaJpZM4J9SiV
.

Contributor

vishh commented Sep 14, 2016

It expects all the cgroups mounts from the host to be mounted. It uses
libcontainer to detect the cgroup mounts.

On Wed, Sep 14, 2016 at 4:10 PM, Mrunal Patel notifications@github.com
wrote:

@vishh https://github.com/vishh Yes, libcontainer looks at the first
mount points that it finds. How is cadvisor expected to be launched in a
container? What host paths are expected to be mounted inside? This will
help identify how to fix this.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKHHJ0rWbzBkuwzXDHd5t49NN_Kxuks5qqH7RgaJpZM4J9SiV
.

@mrunalp

This comment has been minimized.

Show comment
Hide comment
@mrunalp

mrunalp Sep 14, 2016

@vishh with a -v /sys:/sys:ro ?

mrunalp commented Sep 14, 2016

@vishh with a -v /sys:/sys:ro ?

@mrunalp

This comment has been minimized.

Show comment
Hide comment
@mrunalp

mrunalp Sep 14, 2016

@vishh I went through the code and I think it will be easier if we discuss this over a call. What you really want is the later mount point from mountinfo but that isn't always the correct assumption that could be made in libcontainer especially when nesting containers (like docker-in-docker).

mrunalp commented Sep 14, 2016

@vishh I went through the code and I think it will be easier if we discuss this over a call. What you really want is the later mount point from mountinfo but that isn't always the correct assumption that could be made in libcontainer especially when nesting containers (like docker-in-docker).

@vishh

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh Sep 15, 2016

Contributor

sgtm!

On Wed, Sep 14, 2016 at 4:48 PM, Mrunal Patel notifications@github.com
wrote:

@vishh https://github.com/vishh I went through the code and I think it
will be easier if we discuss this over a call. What you really want is the
later mount point from mountinfo but that isn't always the correct
assumption that could be made in libcontainer especially when nesting
containers (like docker-in-docker).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKEZhk1QYdyQfKETWW9Oc35zvIbXMks5qqIfTgaJpZM4J9SiV
.

Contributor

vishh commented Sep 15, 2016

sgtm!

On Wed, Sep 14, 2016 at 4:48 PM, Mrunal Patel notifications@github.com
wrote:

@vishh https://github.com/vishh I went through the code and I think it
will be easier if we discuss this over a call. What you really want is the
later mount point from mountinfo but that isn't always the correct
assumption that could be made in libcontainer especially when nesting
containers (like docker-in-docker).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKEZhk1QYdyQfKETWW9Oc35zvIbXMks5qqIfTgaJpZM4J9SiV
.

smarterclayton added a commit to smarterclayton/origin that referenced this issue Sep 15, 2016

UPSTREAM: google/cadvisor: <drop>: Hack around cgroup load
Running containerized on RHEL7 exposes a cAdvisor bug
google/cadvisor#1461 where the cgroup path
given by docker (cpuacct,cpu) is used in preference to the cgroup path
mounted in with -v /sys:/sys. This commit skips all cgroup paths that
have a comma until we have a real fix in libcontainer and cadvisor.

smarterclayton added a commit to smarterclayton/origin that referenced this issue Sep 15, 2016

UPSTREAM: google/cadvisor: <drop>: Hack around cgroup load
Running containerized on RHEL7 exposes a cAdvisor bug
google/cadvisor#1461 where the cgroup path
given by docker (cpuacct,cpu) is used in preference to the cgroup path
mounted in with -v /sys:/sys. This commit skips all cgroup paths that
have a comma until we have a real fix in libcontainer and cadvisor.
@mrunalp

This comment has been minimized.

Show comment
Hide comment
@mrunalp

mrunalp Sep 15, 2016

I propose that we enhance the libcontainer API to allow getting all the mounts in order. Then it is up to the caller to use them in whatever way they see fit. See opencontainers/runc@5e91f1c cadvisor could just pick the later entries and it should all work fine.

mrunalp commented Sep 15, 2016

I propose that we enhance the libcontainer API to allow getting all the mounts in order. Then it is up to the caller to use them in whatever way they see fit. See opencontainers/runc@5e91f1c cadvisor could just pick the later entries and it should all work fine.

smarterclayton added a commit to smarterclayton/origin that referenced this issue Sep 15, 2016

UPSTREAM: google/cadvisor: <drop>: Hack around cgroup load
Running containerized on RHEL7 exposes a cAdvisor bug
google/cadvisor#1461 where the cgroup path
given by docker (cpuacct,cpu) is used in preference to the cgroup path
mounted in with -v /sys:/sys. This commit skips all cgroup paths that
have a comma until we have a real fix in libcontainer and cadvisor.
@derekwaynecarr

This comment has been minimized.

Show comment
Hide comment
@derekwaynecarr

derekwaynecarr Sep 15, 2016

Collaborator

I spoke w/ @mrunalp and we want to proceed as follows:

Given the ordering of the mount points, cAdvisor when iterating them will get the proper items in its map by virtue of its iteration order in the slice as desired.

Collaborator

derekwaynecarr commented Sep 15, 2016

I spoke w/ @mrunalp and we want to proceed as follows:

Given the ordering of the mount points, cAdvisor when iterating them will get the proper items in its map by virtue of its iteration order in the slice as desired.

smarterclayton added a commit to smarterclayton/origin that referenced this issue Sep 15, 2016

UPSTREAM: google/cadvisor: <drop>: Hack around cgroup load
Running containerized on RHEL7 exposes a cAdvisor bug
google/cadvisor#1461 where the cgroup path
given by docker (cpuacct,cpu) is used in preference to the cgroup path
mounted in with -v /sys:/sys. This commit skips all cgroup paths that
have a comma until we have a real fix in libcontainer and cadvisor.

smarterclayton added a commit to smarterclayton/origin that referenced this issue Sep 16, 2016

UPSTREAM: google/cadvisor: <drop>: Hack around cgroup load
Running containerized on RHEL7 exposes a cAdvisor bug
google/cadvisor#1461 where the cgroup path
given by docker (cpuacct,cpu) is used in preference to the cgroup path
mounted in with -v /sys:/sys. This commit skips all cgroup paths that
have a comma until we have a real fix in libcontainer and cadvisor.
@timstclair

This comment has been minimized.

Show comment
Hide comment
@timstclair

timstclair Sep 16, 2016

Contributor

Do we need this in k8s v1.4 / cAdvisor v0.24? The cAdvisor release was supposed to be cut today, but I held it back. If so, please get approval for cherrypicking into k8s v1.4 first.

Contributor

timstclair commented Sep 16, 2016

Do we need this in k8s v1.4 / cAdvisor v0.24? The cAdvisor release was supposed to be cut today, but I held it back. If so, please get approval for cherrypicking into k8s v1.4 first.

@timstclair

This comment has been minimized.

Show comment
Hide comment
@timstclair

timstclair Sep 16, 2016

Contributor

@pwittrock in case we need this for v1.4

Contributor

timstclair commented Sep 16, 2016

@pwittrock in case we need this for v1.4

@timstclair

This comment has been minimized.

Show comment
Hide comment
@timstclair
Contributor

timstclair commented Sep 16, 2016

@eparis

This comment has been minimized.

Show comment
Hide comment
@eparis

eparis Sep 16, 2016

Contributor

@timstclair I think yes. Which means maybe you get the cadvisor rebase after all :)

Contributor

eparis commented Sep 16, 2016

@timstclair I think yes. Which means maybe you get the cadvisor rebase after all :)

@derekwaynecarr

This comment has been minimized.

Show comment
Hide comment
@derekwaynecarr

derekwaynecarr Sep 16, 2016

Collaborator

I didn't think this needed to gate 1.4 as it only impacts running cAdvisor or Kubelet in a container... /cc @eparis @dawnchen

Collaborator

derekwaynecarr commented Sep 16, 2016

I didn't think this needed to gate 1.4 as it only impacts running cAdvisor or Kubelet in a container... /cc @eparis @dawnchen

@dchen1107

This comment has been minimized.

Show comment
Hide comment
@dchen1107

dchen1107 Sep 16, 2016

Collaborator

Agreed with @derekwaynecarr on that this shouldn't be the blocker for 1.4 release.

Collaborator

dchen1107 commented Sep 16, 2016

Agreed with @derekwaynecarr on that this shouldn't be the blocker for 1.4 release.

guilhermebr added a commit to guilhermebr/origin that referenced this issue Sep 17, 2016

UPSTREAM: google/cadvisor: <drop>: Hack around cgroup load
Running containerized on RHEL7 exposes a cAdvisor bug
google/cadvisor#1461 where the cgroup path
given by docker (cpuacct,cpu) is used in preference to the cgroup path
mounted in with -v /sys:/sys. This commit skips all cgroup paths that
have a comma until we have a real fix in libcontainer and cadvisor.
@philips

This comment has been minimized.

Show comment
Hide comment
@philips

philips Sep 21, 2016

Contributor

@mrunalp What do you mean by "Now, when we bind mount /sys:/sys we are overwriting that with the order on the system which happens to be the opposite on RHEL and hence the issue?"

Contributor

philips commented Sep 21, 2016

@mrunalp What do you mean by "Now, when we bind mount /sys:/sys we are overwriting that with the order on the system which happens to be the opposite on RHEL and hence the issue?"

@derekwaynecarr

This comment has been minimized.

Show comment
Hide comment
@derekwaynecarr

derekwaynecarr Sep 23, 2016

Collaborator

this is fixed now, take a look at the PR and should be clear. If your host
had cpu,cpuacct but inside the container it was always looking for
cpuacct,cpu independent of what the host had since it didn't look at all
mountpoints. So if your OS had sys/fs/cgroup/ setup in the opposite order,
you would hit this.

On Wednesday, September 21, 2016, Brandon Philips <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

@mrunalp https://github.com/mrunalp What do you mean by "Now, when we
bind mount /sys:/sys we are overwriting that with the order on the system
which happens to be the opposite on RHEL and hence the issue?"


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF8dbIcWLsE-AqPrKVOdeZtYEDc0B8scks5qsakEgaJpZM4J9SiV
.

Collaborator

derekwaynecarr commented Sep 23, 2016

this is fixed now, take a look at the PR and should be clear. If your host
had cpu,cpuacct but inside the container it was always looking for
cpuacct,cpu independent of what the host had since it didn't look at all
mountpoints. So if your OS had sys/fs/cgroup/ setup in the opposite order,
you would hit this.

On Wednesday, September 21, 2016, Brandon Philips <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

@mrunalp https://github.com/mrunalp What do you mean by "Now, when we
bind mount /sys:/sys we are overwriting that with the order on the system
which happens to be the opposite on RHEL and hence the issue?"


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1461 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF8dbIcWLsE-AqPrKVOdeZtYEDc0B8scks5qsakEgaJpZM4J9SiV
.

@qband

This comment has been minimized.

Show comment
Hide comment
@qband

qband Oct 9, 2016

Here is the solution that works kubernetes/kubernetes#32728 (comment)

qband commented Oct 9, 2016

Here is the solution that works kubernetes/kubernetes#32728 (comment)

@nnordrum

This comment has been minimized.

Show comment
Hide comment
@nnordrum

nnordrum Oct 13, 2016

Assuming you're running in docker, you can do this too:

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu:rw \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

I just took the example from the front page and added this:

  --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu:rw \

Works like a champ!

nnordrum commented Oct 13, 2016

Assuming you're running in docker, you can do this too:

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu:rw \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

I just took the example from the front page and added this:

  --volume=/sys/fs/cgroup/cpu,cpuacct:/sys/fs/cgroup/cpuacct,cpu:rw \

Works like a champ!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment