daemon mount namespaces #10225

vbatts · 2015-01-20T21:03:31Z

This change covers two of the contrib init's. The daemon ought to be invoked in its own mount namespace, unshared from the root mount namespace.

Some of the error most often seen affect the devicemapper graph driver, seen as EBUSY or "device or resource busy".
Though others are likely to run in to this as folks will have containers that will have mounts or even devices being mounted.

If any other pid outside of the docker daemon unshares their mount namespace, and that pid's mountinfo includes a mount reference of a container, then even when the container exits, and attempts to remove, there kernel will return that the path is busy, due to the mount reference by the outside pid.

I've written about this here: http://blog.hashbangbash.com/2014/11/docker-devicemapper-fix-for-device-or-resource-busy-ebusy/

I did not include a commit for the init scripts with start-stop-daemon, as I would like them to get more testing first. See:

openrc - http://pastebin.com/tT1jUhzR
sysvinit-debian - http://pastebin.com/7hbkSTna

(and for completeness, here is the change for ubuntu's upstart http://pastebin.com/D5MsVCUK)

This systemd.exec setting will construct a new mount namespace for the docker daemon, and use slave shared-subtree mounts so that volume mounts propogate correctly into containers. By having an unshared mount namespace for the daemon it ensures that mount references are not held by other pids outside of the docker daemon. Frequently this can be seen in EBUSY or "device or resource busy" errors. Signed-off-by: Vincent Batts <vbatts@redhat.com>

unshare the mount namespace of the docker daemon to avoid other pids outside the daemon holding mount references of docker containers. Signed-off-by: Vincent Batts <vbatts@redhat.com>

tianon · 2015-01-20T21:05:24Z

LGTM, although I think we should wait until after 1.5 (and then include the start-stop-daemon and upstart changes too)

tianon · 2015-01-20T21:09:02Z

Unless you can make a compelling argument for this not breaking other stuff so close to our freeze window. 😉

vbatts · 2015-01-20T21:11:48Z

https://github.com/docker/docker/issues?q=is%3Aissue+is%3Aopen+%22device+or+resource+busy%22 bothers me

vbatts · 2015-01-20T21:12:31Z

and, this is ./contrib/ ;-)

thaJeztah · 2015-01-20T21:20:57Z

^^ Add a line to those files: "In case of emergency, call Vincent +1 555 ...", then it should be good to merge 😸

vbatts · 2015-01-20T21:30:20Z

@thaJeztah woah there, cowboy ;-)

thaJeztah · 2015-01-20T21:37:39Z

I'm not very familiar with this stuff, but would this also have a positive effect on AUFS not freeing inodes I saw reported?

vbatts · 2015-01-20T21:46:13Z

@thaJeztah do you have a link to that conversation?

thaJeztah · 2015-01-20T22:15:28Z

Yup, read it earlier today, that's why I thought of it; #9755

dqminh · 2015-01-21T07:02:42Z

@vbatts Do you think this will fix aufs device and resource busy bugs too ? I had a few reports about it recently, and the symptoms look familiar.

tianon · 2015-01-21T15:13:35Z

@vbatts For the redhat sysvinit change, I agree 100% that the only repercussions would be to the redhat packagers who are quite familiar with creating out-of-band patches.

For the systemd change, this is used in the packages we release, so I'm much more leery (since a problem there would mean we have to re-release depending on the severity and the number of times it gets reported 😞).

Regardless, we'll need our systemd maintainers to weigh in: @lsm5 @philips @jfrazelle

tianon · 2015-01-21T15:14:37Z

(see also https://github.com/docker/docker/blob/9de604ad68033d08721ce61db7300cee338837e2/project/make/ubuntu#L27-L47)

cpuguy83 · 2015-01-21T15:23:39Z

Didn't @crosbymichael already try doing this, although within Docker itself? #4599

Also, what happens in situations where someone has mounted an fs on the host and wants to then bind-mount that in a container?

unclejack · 2015-01-21T16:37:05Z

@cpuguy83 No, that wasn't merged because there were other problems to take into account.

vbatts · 2015-01-21T16:41:19Z

@tianon well, fedora and centos7 had been using MountFlags=private for a number of versions. So it has had exposure and testing. They have recently switched to MountFlags=slave, for the purpose of ensuring the shared-subtree mounts propagate in the right direction.

tianon · 2015-01-21T20:31:53Z

@vbatts OK, that's fair; so, if the contrib/init/systemd maintainers (@lsm5 @philips @jfrazelle) sign-off on this, I'm OK to merge it for 1.5

I think it's probably prudent that we at least ping @maxamillion for the sysvinit change. 😄

Also, for future reference, the argument that "it's just contrib" doesn't hold any water with me. We use bits of contrib in our releases (as I linked above).

vbatts · 2015-01-21T21:14:49Z

@tianon fair. I like to keep contrib fresh and current too. ❤️

jessfraz · 2015-01-21T21:44:48Z

LGTM

lsm5 · 2015-01-22T00:45:06Z

contrib/init/systemd/docker.service

@@ -6,6 +6,7 @@ Requires=docker.socket

 [Service]
 ExecStart=/usr/bin/docker -d -H fd://
+MountFlags=slave


daemon mount namespaces

crosbymichael · 2015-01-22T01:47:49Z

Y'all know that you can no longer go into /var/lib/docker from the host with this change and inspect/backup a containers filesystems because the mounts happen in a new namespace. And by y'all i mean @vbatts ;)

jessfraz · 2015-01-22T01:48:35Z

I am kinda in favor of possibly reverting this now because there are instances when people need to backup a stopped container etc.

dqminh · 2015-01-22T01:53:10Z

Being able to access current active mounts in /var/lib/docker is currently important for some of our use cases too ( we use aufs, but i guess that would apply to other drivers ), since we do have some tasks that runs outside of container and exposing the container's active filesystem.

vbatts · 2015-01-22T03:12:50Z

works fine on overlayfs even. I don't know what to say for aufs. Really a shame that you'd improve many folks experience, reduce errors, but determine a need to revert because of mystical behavior on aufs.

jessfraz · 2015-01-22T03:16:10Z

Is there another solution maybe :(

On Wednesday, January 21, 2015, Vincent Batts notifications@github.com
wrote:

works fine on overlayfs even. I don't know what to say for aufs. Really a
shame that you'd improve many folks experience, reduce errors, but
determine a need to revert because of mystical behavior on aufs.

—
Reply to this email directly or view it on GitHub
#10225 (comment).

vbatts · 2015-01-22T03:28:00Z

@jfrazelle i even tried https://gist.github.com/vbatts/7257da4823c8bc8c61c1 and the directories are still hidden on aufs.

jessfraz · 2015-01-22T03:29:56Z

shakes fist in air ok @crosbymichael you think we should revert?

On Wednesday, January 21, 2015, Vincent Batts notifications@github.com
wrote:

@jfrazelle https://github.com/jfrazelle i even tried
https://gist.github.com/vbatts/7257da4823c8bc8c61c1 and the directories
are still hidden on aufs.

—
Reply to this email directly or view it on GitHub
#10225 (comment).

jessfraz · 2015-01-22T03:32:16Z

We could also add something to docs about how this recommended for
devicemapper

On Wednesday, January 21, 2015, Jessica Frazelle jess@docker.com wrote:

shakes fist in air ok @crosbymichael you think we should revert?

On Wednesday, January 21, 2015, Vincent Batts <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

@jfrazelle https://github.com/jfrazelle i even tried
https://gist.github.com/vbatts/7257da4823c8bc8c61c1 and the directories
are still hidden on aufs.

—
Reply to this email directly or view it on GitHub
#10225 (comment).

dqminh · 2015-01-22T04:01:13Z

I think the PR is safe as-is right ? It only modifies systemd / sysvinit-redhat init script and aufs is not generally used on those systems while devicemapper and btrfs are.

It's probably better to keep this and add some docs on how this doesnt work with aufs.

jessfraz · 2015-01-22T04:23:54Z

I have debian aufs machines, actually the core machine is debian aufs.

On Wednesday, January 21, 2015, Daniel, Dao Quang Minh <
notifications@github.com> wrote:

I think the PR is safe as-is right ? It only modifies systemd /
sysvinit-redhat init script and aufs is not generally used on those systems
while devicemapper and btrfs are.

It's probably better to keep this and add some docs on how this doesnt
work with aufs.

—
Reply to this email directly or view it on GitHub
#10225 (comment).

dqminh · 2015-01-22T04:28:57Z

@jfrazelle debian should use sysvinit-debian script, which is not affected by this change right ?

jessfraz · 2015-01-22T04:34:02Z

Not debian jessie ;)

On Wednesday, January 21, 2015, Daniel, Dao Quang Minh <
notifications@github.com> wrote:

@jfrazelle https://github.com/jfrazelle debian should use
sysvinit-debian script, which is not affected by this change right ?

—
Reply to this email directly or view it on GitHub
#10225 (comment).

jessfraz · 2015-01-22T04:34:51Z

As much as I would love to be the only debian jessie user running aufs I do
not think that is the case

On Wednesday, January 21, 2015, Daniel, Dao Quang Minh <
notifications@github.com> wrote:

@jfrazelle https://github.com/jfrazelle debian should use
sysvinit-debian script, which is not affected by this change right ?

—
Reply to this email directly or view it on GitHub
#10225 (comment).

jessfraz · 2015-01-22T04:35:36Z

Also what about arch linux users with aufs

On Wednesday, January 21, 2015, Jessica Frazelle jess@docker.com wrote:

Not debian jessie ;)

On Wednesday, January 21, 2015, Daniel, Dao Quang Minh <
notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

@jfrazelle https://github.com/jfrazelle debian should use
sysvinit-debian script, which is not affected by this change right ?

—
Reply to this email directly or view it on GitHub
#10225 (comment).

dqminh · 2015-01-22T04:35:51Z

oh noes 🔥

dqminh · 2015-01-22T07:56:23Z

@vbatts weird, https://gist.github.com/vbatts/7257da4823c8bc8c61c1 works on my aufs system.

# without setns
> ls /var/lib/docker/aufs/mnt/f714499277a27743ad55e052d4ab1f7946a8baf98a64463e521aa9a0f25f242f

# with setns, 30411 is docker daemon pid
> gcc -o test-setns main.c
> sudo ./test-setns /proc/30411/ns/mnt ls /var/lib/docker/aufs/mnt/f714499277a27743ad55e052d4ab1f7946a8baf98a64463e521aa9a0f25f242f
bin  dev  etc  file.txt  home  lib  lib64  linuxrc  media  mnt  opt  proc  root  run  sbin  sys  tmp  usr  var

vbatts · 2015-01-22T14:48:14Z

@dqminh oh good. is that on a running or stopped container?

Perhaps this is the start of a contrib tool for debugging containers in their namespace.

dqminh · 2015-01-22T15:00:58Z

@vbatts it's on a running container. If the container is stopped, then /var/lib/docker/aufs/mnt/container-id should be empty right, because it's unmounted.

philips · 2015-01-22T18:47:34Z

@dqminh Now I am confused. Does your version of aufs differ from @jfrazelle 's?

crosbymichael · 2015-01-22T18:48:52Z

Running containers, we don't leave them mounted if they are stopped

vbatts · 2015-01-22T18:49:47Z

#10281 is up for debate

http://blog.hashbangbash.com/2014/11/docker-devicemapper-fix-for-device-or-resource-busy-ebusy/ moby/moby#10225 Also, we'll need an extra option in the future - commented out now, uncomment when it's introduced (probably Docker 1.7.0).

tianon · 2015-04-17T22:46:39Z

@vbatts heh, looks like we forgot to push this over to sysvinit, upstart, and openrc too

vbatts · 2015-04-20T13:21:26Z

@tianon

6bb6586

is one, but the others were hackish, with start-stop-daemon. I may have a branch floating around here though...

tianon · 2015-04-20T18:34:18Z

Yeah, I remember that we discussed it quite a bit and thought we'd found a solution to the start-stop-daemon problems. 😢

tianon · 2015-04-21T21:49:13Z

https://botbot.me/freenode/docker-maintainers/msg/30055754/

vbatts · 2015-04-21T22:28:41Z

"I want to die" that was the solution there
On Apr 21, 2015 5:49 PM, "Tianon Gravi" notifications@github.com wrote:

https://botbot.me/freenode/docker-maintainers/msg/30055754/

—
Reply to this email directly or view it on GitHub
#10225 (comment).

vbatts added 2 commits January 20, 2015 14:22

contrib/sysvinit-redhat: unshare mount namespace

6bb6586

unshare the mount namespace of the docker daemon to avoid other pids outside the daemon holding mount references of docker containers. Signed-off-by: Vincent Batts <vbatts@redhat.com>

vbatts force-pushed the vbatts-init_mount_namespaces branch from b2f2aea to 6bb6586 Compare January 20, 2015 21:03

unclejack mentioned this pull request Jan 21, 2015

Try to cleanly unmount docker block devices #10229

Closed

unclejack added this to the 1.5.0 milestone Jan 21, 2015

unclejack mentioned this pull request Jan 21, 2015

Failed to remove instance as device is busy #10174

Closed

lsm5 reviewed Jan 22, 2015
View reviewed changes

jessfraz pushed a commit that referenced this pull request Jan 22, 2015

Merge pull request #10225 from vbatts/vbatts-init_mount_namespaces

fcc4abc

daemon mount namespaces

jessfraz merged commit fcc4abc into moby:master Jan 22, 2015

dqminh mentioned this pull request Jan 23, 2015

Revert "contrib/systemd: mount namespace and subtree flags" #10281

Closed

SvenDowideit mentioned this pull request Jan 27, 2015

Post 1.4.1 docs update 2 #9739

Merged

thaJeztah mentioned this pull request Jul 3, 2015

Docker breaks autofs directory in V1.5, used to work in V1.3.3 #13376

Closed

thaJeztah mentioned this pull request Mar 8, 2016

Shared subtrees not working under Debian #19625

Closed

vbatts deleted the vbatts-init_mount_namespaces branch April 27, 2016 18:03

jhaynes mentioned this pull request May 12, 2017

Mounting EBS Volumes requires ECS Agent and Docker restart aws/amazon-ecs-agent#799

Closed

daemon mount namespaces #10225

daemon mount namespaces #10225

Uh oh!

Conversation

vbatts commented Jan 20, 2015

Uh oh!

tianon commented Jan 20, 2015

Uh oh!

tianon commented Jan 20, 2015

Uh oh!

vbatts commented Jan 20, 2015

Uh oh!

vbatts commented Jan 20, 2015

Uh oh!

thaJeztah commented Jan 20, 2015

Uh oh!

vbatts commented Jan 20, 2015

Uh oh!

thaJeztah commented Jan 20, 2015

Uh oh!

vbatts commented Jan 20, 2015

Uh oh!

thaJeztah commented Jan 20, 2015

Uh oh!

dqminh commented Jan 21, 2015

Uh oh!

tianon commented Jan 21, 2015

Uh oh!

tianon commented Jan 21, 2015

Uh oh!

cpuguy83 commented Jan 21, 2015

Uh oh!

unclejack commented Jan 21, 2015

Uh oh!

vbatts commented Jan 21, 2015

Uh oh!

tianon commented Jan 21, 2015

Uh oh!

vbatts commented Jan 21, 2015

Uh oh!

jessfraz commented Jan 21, 2015

Uh oh!

lsm5 Jan 22, 2015

Choose a reason for hiding this comment

Uh oh!

crosbymichael commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

dqminh commented Jan 22, 2015

Uh oh!

vbatts commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

vbatts commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

dqminh commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

dqminh commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

jessfraz commented Jan 22, 2015

Uh oh!

dqminh commented Jan 22, 2015

Uh oh!

dqminh commented Jan 22, 2015

Uh oh!

vbatts commented Jan 22, 2015

Uh oh!

dqminh commented Jan 22, 2015