Shared subtrees not working under Debian

[lox]

I'm having issues getting the new --volume /mnt/shared:/shared:shared feature of 1.10 working (from #17034).

root@testhost:~# uname -a
Linux testhost 3.18.5-031805-generic #201501292218 SMP Fri Jan 30 03:19:17 UTC 2015 x86_64 GNU/Linux

root@testhost:~# docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 9
Server Version: 1.10.0-dev
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: host bridge null
Kernel Version: 3.18.5-031805-generic
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 1.899 GiB
Name: loxflix
ID: RNAT:QYNF:66KE:DSN5:NRPC:ITAA:6G3M:WMR2:JABB:CPGO:ZN7O:FAVU
Debug mode (server): true
 File Descriptors: 13
 Goroutines: 31
 System Time: 2016-01-24T08:45:00.139001319Z
 EventsListeners: 0
 Init SHA1: b84242d186971c8111d1f9de77b7c476bc049614
 Init Path: /usr/lib/docker/dockerinit
 Docker Root Dir: /var/lib/docker
Labels:
 provider=generic
Experimental: true

root@testhost:~# findmnt -o TARGET,PROPAGATION /mnt/testhost
TARGET       PROPAGATION
/mnt/shared shared

root@testhost:~# docker run --rm -it --volume /mnt/shared:/data:shared ubuntu:14.04
docker: Error response from daemon: Cannot start container 6859ea4fc2f23130da7e72f301c9a82528c6e78e406b6f68261d000410ad6960: Path /mnt/shared is mounted on /mnt/shared but it is not a shared mount..

I had this working under ubuntu. One of the key differences I noted was that mount indicated that the mount /shared had -o bind where as under debian, it doesn't seem to.

[GordonTheTurtle]

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

For more information about reporting issues, see CONTRIBUTING.md.

You don't have to include this information if this is a feature request

(This is an automated, informational response)

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

[lox]

Initially I tried this under Debian 8.2, with a 3.18.5, but subsequently upgraded to Debian stretch, with a 4.3x kernel. Same issue.

[cpuguy83]

@lox The dir being shared needs to be flagged as shared as well (or it's parent needs to be shared).

[lox]

Brian, I've spent a considerable time investigating this and reading the
various PRs and the sub tree docs, I understand that you get a lot of
spurious reports, and perhaps mine is too, but it would be good to verify
that the fix is as you expect before closing the ticket.

I'm trying to parse your response, do you mean the original mount point? If
so, the output of findmnt is showing /mnt/shared as shared. I've tried
bound /mnt/shared to itself as it's simply a dir on the main partition and
then called mount --make-shared /mnt/shared before invoking docker.

I've also tried mount --make-shared / with no effect. Where in the code
does docker determine the options set on a mount? My theory is its
correctly flagged as shared but for some reason docker isn't seeing that.

As I said, I've had this working fine under several Ubuntu installations.

On Sunday, January 24, 2016, Brian Goff notifications@github.com wrote:

Closed #19625 #19625.

—
Reply to this email directly or view it on GitHub
#19625 (comment).

[cpuguy83]

Consider this example:

$ mkdir /foo
$ mount -o bind /foo /foo
$ mount --make-shared /foo
$ docker run -v /foo:/foo:rshared busybox sh

This should work.
If it's not working, I'd suspect something about the mount that is actually at /mnt/shared?

[cpuguy83]

Here is the check:
https://github.com/docker/docker/blob/4ee3048fa8382f9e9af2418029b8e53885bb906a/daemon/execdriver/native/create.go#L348

[lox]

Ok, so have a brand new Debian Stretch system. This is what /proc/self/mountinfo (via https://github.com/docker/docker/blob/4ee3048fa8382f9e9af2418029b8e53885bb906a/pkg/mount/mount.go) looks like:

root@lado1:~# cat /proc/self/mountinfo
16 21 0:16 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
17 21 0:4 / /proc rw,nosuid,nodev,noexec,relatime shared:12 - proc proc rw
18 21 0:6 / /dev rw,relatime shared:2 - devtmpfs udev rw,size=10240k,nr_inodes=242870,mode=755
19 18 0:13 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
20 21 0:17 / /run rw,nosuid,relatime shared:5 - tmpfs tmpfs rw,size=398984k,mode=755
21 0 202:0 / / rw,relatime shared:1 - ext4 /dev/xvda rw,errors=remount-ro,stripe=32,data=ordered
22 16 0:18 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
23 18 0:19 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw
24 20 0:20 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
25 16 0:21 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755
26 25 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd
27 16 0:23 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:11 - pstore pstore rw
28 25 0:24 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,net_cls,net_prio
29 25 0:25 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,freezer
30 25 0:26 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,cpu,cpuacct
31 25 0:27 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,perf_event
32 25 0:28 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,devices
33 25 0:29 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,blkio
34 25 0:30 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,cpuset
35 17 0:31 / /proc/sys/fs/binfmt_misc rw,relatime shared:20 - autofs systemd-1 rw,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct
36 18 0:15 / /dev/mqueue rw,relatime shared:21 - mqueue mqueue rw
38 16 0:7 / /sys/kernel/debug rw,relatime shared:22 - debugfs debugfs rw
37 18 0:32 / /dev/hugepages rw,relatime shared:23 - hugetlbfs hugetlbfs rw
80 20 0:34 / /run/user/0 rw,nosuid,nodev,relatime shared:64 - tmpfs tmpfs rw,size=199492k,mode=700

Installing docker via curl -fsSL https://test.docker.com/ | sh. Ran your above test case:

root@lado1:~# mkdir /foo
root@lado1:~# mount -o bind /foo /foo
root@lado1:~# mount --make-shared /foo
root@lado1:~# docker run -v /foo:/foo:rshared busybox sh
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
eeee0535bf3c: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:c1bc9b4bffe665bf014a305cc6cf3bca0e6effeb69d681d7a208ce741dad58e0
Status: Downloaded newer image for busybox:latest
docker: Error response from daemon: Cannot start container 9a52fa7730d8d811cf3c3e0f305e875c6b1ac383027f561741a5e8041917d20f: Path /foo is mounted on /foo but it is not a shared mount..

[lox]

The line from /proc/self/mountinfo for /foo is:

108 21 202:0 /foo /foo rw,relatime shared:1 - ext4 /dev/xvda rw,errors=remount-ro,stripe=32,data=ordered

[lox]

Checked the parsing from https://github.com/docker/docker/blob/729c9a97822ebee2c978a322d37060454af6bc66/pkg/mount/mountinfo_linux.go:

{
  "ID": 108,
  "Parent": 21,
  "Major": 202,
  "Minor": 0,
  "Root": "/foo",
  "Mountpoint": "/foo",
  "Opts": "rw,relatime",
  "Optional": "shared:1",
  "Fstype": "ext4",
  "Source": "/dev/xvda",
  "VfsOpts": "rw,errors=remount-ro,stripe=32,data=ordered"
}

The original code you linked to it seems like this should be reading the correct value from mountinfo.Optional. I'm stumped.

[lox]

Of course it works just fine when I build a binary from master.

[lox]

Hmmm, but it doesn't work when running under systemd.

[lox]

Ok, I've verified that with the latest release the problem occurs when docker daemon is running under systemd, but not when running in the foreground. Perhaps some issue with the strange things that systemd does with cgroups and the debian kernel @cpuguy83?

[cpuguy83]

What are the MountFlags set to in your unit file for Docker?

[lox]

Hah, just got there via #12544. It was the default service installed by the docker package, which has MountFlags=slave. When I changed it to shared your example works.

Is this expected behaviour?

[cpuguy83]

@lox yes, that's expected when MountFlags=slave since that changes the root mount to slave instead of shared.

[lox]

It would have been expected if I'd set it thus, but it was the default. Sounds like it would be at least worth adding to the documentation? I imagine lots of people are going to end up wanting to run FUSE filesystems with the subtree stuff and will get very confused.

[cpuguy83]

I think that makes sense if there's nothing along those lines for the systemd case.

[errordeveloper]

Looks like Docker Machine will give you MountFlags=slave as well.

[thaJeztah]

@errordeveloper if you have ideas what needs to be added to the documentation, could you open a pull request?

[errordeveloper]

@thaJeztah hm, I think this is just a bug in Docker Machine, which I'm going to report to and provide a PR, although had there been any reason for this setting to appear in the first place? Which is the very first unit file where this appears?

[errordeveloper]

@thaJeztah ping.

[thaJeztah]

@errordeveloper looks like MountFlags=slave was first added in Docker 1.5.0; see #10225

[jpetazzo]

I hit exactly the same issue (engine 1.10.3 on Ubuntu 14.04 LTS).

I did:

• set MountFlags=shared in the systemd unit
• systemctl daemon-reload
• systemctl restart docker
• mkdir /foo
• mount -o bind /foo /foo
• mount --make-rshared /foo
• docker run -ti -v /foo:/foo:shared busybox

... and I still get docker: Error response from daemon: Cannot start container 430aef7fc5457544fca2419daa7301696b5e0bd2d1e4993ea131ac0a90e3242e: Path /foo is mounted on /foo but it is not a shared mount..

Before I open a separate issue (since this disturbingly to be the same thing), does anybody know a way to double-check the mount flags?

[jpetazzo]

(Confirming that the problem is caused by systemd, since running the Engine manually instantly fixes the issue.)

[jpetazzo]

Also:

root@node1:~# grep Mount /lib/systemd/system/docker.service
MountFlags=shared
root@node1:~# nsenter --mount=/proc/$(cat /var/run/docker.pid)/ns/mnt findmnt -o PROPAGATION /
PROPAGATION
private,slave

🍑

[jpetazzo]

If anybody runs into this later, here are a few useful commands.

1. Check which unit file systemd is currently using:

systemctl status docker | grep Loaded

(In my case, one had been installed into /etc but I was still editing the system-wide one in /lib)

2. Make sure that MountFlags is shared or absent from the unit file

3. Make sure that your unit file doesn't enable PrivateTmp=, PrivateDevices=, ProtectSystem=, ProtectHome=, ReadOnlyDirectories=, InaccessibleDirectories= or ReadWriteDirectories= (because those will automatically switch the MountFlags to slave). See systemd docs.

4. Check the status of your mounts in the namespace of the Engine itself:

nsenter --mount=/proc/$(cat /var/run/docker.pid)/ns/mnt findmnt -o TARGET,PROPAGATION

(This allowed me to confirm that it was set to private,slave instead of shared)

5. Compare the namespaces used for the system and the Engine:

ls -l /proc/1/ns/ /proc/$(cat /var/run/docker.pid)/ns/

(This allowed me to confirm that the Engine was in its own namespace.)

I hope this helps!

[mikedanese]

You can override the MountFlags value in the default unit with a systemd drop-in directory. e.g.:

mkdir -p /etc/systemd/system/docker.service.d/
cat <<EOF > /etc/systemd/system/docker.service.d/clear_mount_propagtion_flags.conf
[Service]
MountFlags=shared
EOF

so you don't have to modify the default unit, which might get reset during a docker upgrade.

[errordeveloper]

Hey folks, can this issue be re-opened and considered for the next release?? I think here is enough activity, it's definitely a problem for anyone trying to use the mount propagation feature, do people agree?

[thaJeztah]

@errordeveloper can you open a pull request that changes the systemd unit file; https://github.com/docker/docker/blob/master/contrib/init/systemd/docker.service#L14, then we can discuss on that PR. (I'm not sure if there's side effects to that change that need to be discussed)

[errordeveloper]

@thaJeztah cheers, here it is #22806!

[davidacce]

sudo mount --make-shared /