provides a new flag to make rootfs all sharable|slave|private propagation settable #77

rootfs · 2015-07-01T15:56:35Z

This is a port of docker-archive/libcontainer#632

Current a container has to nsenter the host's mount namespace to mount filesystem and
share with other containers. This approach doesn't work if the filesystem mount
calls helper utility (/sbin/mount.XXX). This limitation makes containerized kubelet unable to mount certain filesystems.

This commit provides a new flag to make rootfs sharable. Since moving a shared rootfs is semantically confusing for pivot_root(2) and MS_MOVE. A new function changeRoot() is provided to switch rootfs to new destination.

mrunalp · 2015-07-01T22:46:46Z

libcontainer/configs/config.go

+	// "private": rootfs is mounted as MS_PRIVATE
+	// "shared": rootfs is mounted as MS_SHARED
+	// "slave": rootfs is mounted as MS_SLAVE
+	RootMount string `json:"root_mount"`


Could make this more strongly typed like Namespaces instead of strings.

Also, how about calling it RootfsMountMode?

Agree on both.

rhatdan · 2015-07-13T15:48:38Z

@rootfs Could you change this like @mrunalp suggested

"Could make this more strongly typed like Namespaces instead of strings. ... how about calling it RootfsMountMode?"

rootfs · 2015-07-13T15:53:21Z

@rhatdan rootfs@d30dec2

mrunalp · 2015-07-13T20:06:59Z

libcontainer/configs/mount_propagation.go

@@ -0,0 +1,4 @@
+// mount_propagation


Package names are preferred without underscores. https://blog.golang.org/package-names

kelseyhightower · 2015-07-17T15:36:52Z

Is there a chance this will land before Docker 1.8?

LK4D4 · 2015-07-17T15:41:20Z

libcontainer/configs/mountpropagation_linux.go

@@ -0,0 +1,11 @@
+// +build linux


this isn't build tag, it should be separated from package by empty line.
Better just remove it, you use postfix filename notation anyway.

LK4D4 · 2015-07-17T15:53:18Z

Godeps/_workspace/src/github.com/opencontainers/specs/spec_linux.go

@@ -28,6 +28,8 @@ type Linux struct {
 	Capabilities []string `json:"capabilities"`
 	// Devices are a list of device nodes that are created and enabled for the container.
 	Devices []string `json:"devices"`
+	// RootfsPropagation is the rootfs mount propagation mode for the container.
+	RootfsPropagation string `json:"rootfsPropagation"`


I have concerns how this is appeared here without updating Godeps.json.

there is a separate PR against specs

I merged it. Would you mind to update properly?

kelseyhightower · 2015-07-17T16:03:08Z

@rootfs This is so close to the finish line, let me know how I can help. I'm happy to review and test.

LK4D4 · 2015-07-17T16:05:48Z

@kelseyhightower Help with testing always appreciated :)

rootfs · 2015-07-17T16:29:44Z

@kelseyhightower many thanks for helping. Let me know if you are able to run my runc branch with this config.json

mrunalp · 2015-07-17T16:39:38Z

spec.go

+	if p, exists := mountPropagationMapping[spec.Linux.RootfsPropagation]; exists {
+		config.RootfsMountMode = p
+	} else {
+		return nil, fmt.Errorf("invalid rootfs propagation mode:", spec.Linux.RootfsPropagation)


I think we should default to something when it isn't specified.

sure, which default to use?

LK4D4 · 2015-07-18T05:23:53Z

I wonder where is jenkins :)

mrunalp · 2015-07-18T18:54:44Z

rprivate is causing issues with the chmod on the console.

rootfs · 2015-07-19T15:05:34Z

@mrunalp what are the issues? I've tested chmod with rprivate, so far so good. Thanks.

mrunalp · 2015-07-20T16:33:31Z

@rootfs Actually, it can't find the console for me. I will look more.

FATA[0000] Container start failed: chmod /dev/pts/4: no such file or directory

ibuildthecloud · 2015-08-03T14:39:25Z

My concern is if a container mounts a path that is not under bind mount. Won't that cause issues. Again I'm probably missing something here, so sorry if that's a stupid question. I'll try out this patch later today so I can understand it more.

rootfs · 2015-08-03T14:44:57Z

@ibuildthecloud bind mount is probably the only place containers can create new mountpoints that also share with host.

I also have a docker patch based on this PR

rootfs · 2015-08-03T16:09:14Z

Background information about propagation modes and their use cases can be found in [1] and [2]

Propagation modes: Propogate container's mountpoint to the host docker-archive/libcontainer#632 (comment)
Use Cases: Propogate container's mountpoint to the host docker-archive/libcontainer#632 (comment)

mrunalp · 2015-08-03T16:57:35Z

@rootfs I see container mounts leaked on host after container exit when testing "shared"

....
cgroup on /busybox/sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /busybox/sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /busybox/sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /busybox/sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /busybox/sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /busybox/sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /busybox/sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /busybox/sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
proc on /busybox/proc/sys type proc (rw,relatime)
proc on /busybox/proc/sys type proc (rw,relatime)
proc on /busybox/proc/sysrq-trigger type proc (rw,relatime)
proc on /busybox/proc/sysrq-trigger type proc (rw,relatime)
proc on /busybox/proc/irq type proc (rw,relatime)
proc on /busybox/proc/irq type proc (rw,relatime)
proc on /busybox/proc/bus type proc (rw,relatime)
proc on /busybox/proc/bus type proc (rw,relatime)
tmpfs on /busybox/proc/kcore type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755)
tmpfs on /busybox/proc/kcore type tmpfs (rw,nosuid,seclabel,size=65536k,mode=755)
....

rhvgoyal · 2015-08-03T17:58:15Z

Was just chatting with dan walsh and he said that we probably want to make sure that "shared, slave, private" becomes the property of volume being mounted inside the container. (And container root remains PRIVATE). That means only select volumes and any mount under those will become shared and user can control those instead of all of the mounts under container root being shared.

rhatdan · 2015-08-03T18:22:59Z

Right, I believe / should be shared by default (Or arguably slave.) But everything under $rootfs would be PRIVATE. I think the internal mounts should all be done privately. Volume mounts from the host should be controlled on a individual basis, but the default would be shared.

rhvgoyal · 2015-08-03T18:26:08Z

If / is SLAVE then I don't think volumes can be shared. May be we can scan the mount config and if any of the volumes are shared, then use SHARED for /, if any of the volumes are slave, then use
SLAVE for root otherwise fall back to PRIVATE.

rootfs · 2015-08-03T18:27:27Z

@mrunalp somehow in docker there is no leak :)
@rhvgoyal i am not sure if we can use the same parent mount namespace for multiple containers. A new mount namespace doesn't have this problem.
@rhatdan I think if we can make rootfs shared, then directory that are direct under rootfs are propagated. We can make 1st level mountpoint slave/private so 2nd level mountpoint are not propagated.

mrunalp · 2015-08-03T18:28:46Z

@rootfs Docker uses its own mount namespace and so might not be visible in the default namespace.

rhatdan · 2015-08-03T18:31:05Z

@mrunalp Right we need to address that issue also. But I want to just get to the point that we could share mounts between containers.

rhatdan · 2015-08-03T18:35:11Z

@rootfs Right I want to rprivate on rootfs not on /. Bottom line we want to allow volume mounts differently then we do internal mounts.

rootfs · 2015-08-03T18:36:57Z

@rhatdan got it, let me try it, thanks.

rhvgoyal · 2015-08-03T18:42:05Z

Right, once we have the mechanism to have some mounts as "shared" between container and daemon we can look into not running docker daemon in a separate namespace. (Or train users to
get into docker namespace).

rootfs · 2015-08-03T19:03:41Z

@rhatdan new commit makes rootfs private if / is shared.
@mrunalp no more leak

…tion mode settable Signed-off-by: Huamin Chen <hchen@redhat.com>

Signed-off-by: Huamin Chen <hchen@redhat.com>

mrunalp · 2015-08-03T19:56:43Z

@rootfs You could take out the Godep update code as we just merged a PR to update the spec. See #173

rootfs · 2015-08-03T19:57:28Z

@mrunalp cool, thanks. just rebased.

rhvgoyal · 2015-08-03T20:47:42Z

I have taken @rhatdan paches to allow shared/slave volume mounts in a PRIVATE rootfs. Modified it a bit to allow shared mounts. While I can make a volume mount sharable but it does not seem to propagating to host namespace. I am not sure why. I am trying to debug it.

In the mean time, here are the two commits. Hoping somebody can notice what am I doing wrong.

rhvgoyal@cd9a387
rhvgoyal@5cdc224

rhvgoyal · 2015-08-03T22:23:24Z

I think this notion that container rootfs can be private and it can still have mounts under it which are receiving events from host namespace (slave) or sending back events to host namespace (shared) is flawed. Once you make rootfs private, looks like any connection from root.

For example, I created a directory rootfs which contains container root fs. and I did following.

$ unshare -m
$ mount --bind rootfs rootfs
Above mount operation becomes visible in host as well.
$ mount --make-private rootfs
This breaks sharing. Now any mount operation under rootfs, does not get propagated to host namespace.

So I think after clone operation we will have to make sure rootfs is SHARED to make sure events propagate back to hostnamespace.

EDIT:

Looks like above is true only for one level (for certain type of mounts). We can still have mounts deeper in the hierarchy which are shared. For example, I tried following.

$ unshare -m
$mount --bind rootfs rootfs
(Above mount point becomes visible in host as well)
$ mount --make-private rootfs
$ mkdir rootfs/mnt1
$ mkdir mnt1-source
$ mount --bind mnt1-source rootfs/mnt
(This mount operation is not visible in host as parent mount is PRIVATE. Note mnt1 is SHARED though. Not sure where default propagation of a mount point comes from.)
$ mkdir rootfs/mnt1/mnt1child
$ mkdir mnt1-child-source
$ mount --bind mnt1-child-source rootfs/mnt1/mnt1child
(This mount operation does propagate to host namespace as parent mount mnt1 is SHARED).

In summary, rootfs was PRIVATE so any mounts directly under rootfs did not propagate to host. But children of rootfs themselves were SHARED so grand children mounts of rootfs did propagate back to host. So looks like PRIVATE (non-recursive) is effective only for immediate children.

Having said that, it was true only if mount source was outside above rootfs/. If I did convert some directories under rootfs to mount points by bind mounting these over themselves, then propagation did not happen to host and every child mount point was PRIVATE.

rootfs · 2015-08-03T22:30:53Z

Did you run the tests in runc or docker?

rhvgoyal · 2015-08-03T22:32:23Z

@rootfs

I was playing with runc. In above example, I am not using runc or docker. Just trying to make use of "unshare -m" to get a separate mount namespace and playing with mount operations and see what is being propagated back to host mount namespace.

rootfs · 2015-08-03T22:46:50Z

@rhatdan pointed out there were two rootfs: one that lives on the host and is where container's / binds to, the other is container's /. Making the first rootfs private is ok, because it blocks container's /dev, /proc, etc from showing up on the host. As long as container's / (i.e. second rootfs) is shared, mounts created under container's / is propagated to host.

Signed-off-by: Huamin Chen <hchen@redhat.com>

rootfs · 2015-08-17T22:02:15Z

closed. please comment on the new PR #208

runtime.md: convert oc to runc

LK4D4 mentioned this pull request Jul 1, 2015

Propogate container's mountpoint to the host docker-archive/libcontainer#632

Closed

mrunalp reviewed Jul 1, 2015
View reviewed changes

rootfs force-pushed the dev branch from 976f94a to d30dec2 Compare July 6, 2015 15:12

mrunalp mentioned this pull request Jul 6, 2015

It should be possible to set the mount propagation modes for containers and volumes opencontainers/runtime-spec#56

Closed

mrunalp reviewed Jul 13, 2015
View reviewed changes

GordonTheTurtle added the dco/no label Jul 14, 2015

rootfs force-pushed the dev branch from b31c9da to b3c64a7 Compare July 14, 2015 15:43

This was referenced Jul 14, 2015

Experiment run kubelet as docker container kubernetes/kubernetes#4869

Closed

It should be possible to set the propagation mode of a container's mnt namespace moby/moby#14630

Closed

rootfs force-pushed the dev branch from b3c64a7 to ef2fc9c Compare July 14, 2015 17:50

alban mentioned this pull request Jul 15, 2015

support mount propagation modes rkt/rkt#1149

Open

rootfs force-pushed the dev branch from ef2fc9c to b68acd7 Compare July 15, 2015 18:25

jsafrane mentioned this pull request Jul 17, 2015

RFC: Use containers to mount Kubernetes volumes kubernetes/kubernetes#10176

Closed

kelseyhightower mentioned this pull request Jul 17, 2015

Update libcontainer moby/moby#14693

Merged

LK4D4 reviewed Jul 17, 2015
View reviewed changes

rootfs force-pushed the dev branch from 7af6006 to 4feed9e Compare July 17, 2015 15:43

LK4D4 reviewed Jul 17, 2015
View reviewed changes

mrunalp reviewed Jul 17, 2015
View reviewed changes

rootfs added 2 commits August 3, 2015 15:52

provides a new flag to make rootfs all sharable|slave|private propaga…

165cfe0

…tion mode settable Signed-off-by: Huamin Chen <hchen@redhat.com>

review feedback: if / is shared, make rootfs private

98e63f8

Signed-off-by: Huamin Chen <hchen@redhat.com>

rootfs force-pushed the dev branch from fdddb0c to 98e63f8 Compare August 3, 2015 19:52

fix rebase

7c50349

Signed-off-by: Huamin Chen <hchen@redhat.com>

rootfs referenced this pull request in rootfs/runc Aug 10, 2015

if mnt uses host namespace, just chroot()

c356086

Signed-off-by: Huamin Chen <hchen@redhat.com>

rootfs closed this Aug 17, 2015

stefanberger pushed a commit to stefanberger/runc that referenced this pull request Sep 8, 2017

Merge pull request opencontainers#77 from laijs/convert-oc-to-runc

e92fa18

runtime.md: convert oc to runc

provides a new flag to make rootfs all sharable|slave|private propagation settable #77

provides a new flag to make rootfs all sharable|slave|private propagation settable #77

Conversation

rootfs commented Jul 1, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhatdan commented Jul 13, 2015

rootfs commented Jul 13, 2015

Choose a reason for hiding this comment

kelseyhightower commented Jul 17, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kelseyhightower commented Jul 17, 2015

LK4D4 commented Jul 17, 2015

rootfs commented Jul 17, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LK4D4 commented Jul 18, 2015

mrunalp commented Jul 18, 2015

rootfs commented Jul 19, 2015

mrunalp commented Jul 20, 2015

ibuildthecloud commented Aug 3, 2015

rootfs commented Aug 3, 2015

rootfs commented Aug 3, 2015

mrunalp commented Aug 3, 2015

rhvgoyal commented Aug 3, 2015

rhatdan commented Aug 3, 2015

rhvgoyal commented Aug 3, 2015

rootfs commented Aug 3, 2015

mrunalp commented Aug 3, 2015

rhatdan commented Aug 3, 2015

rhatdan commented Aug 3, 2015

rootfs commented Aug 3, 2015

rhvgoyal commented Aug 3, 2015

rootfs commented Aug 3, 2015

mrunalp commented Aug 3, 2015

rootfs commented Aug 3, 2015

rhvgoyal commented Aug 3, 2015

rhvgoyal commented Aug 3, 2015

rootfs commented Aug 3, 2015

rhvgoyal commented Aug 3, 2015

rootfs commented Aug 3, 2015

rootfs commented Aug 17, 2015