New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs docker changed or maybe fixed the volume issue in 1.12 ? #25474

Closed
kouhin opened this Issue Aug 7, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@kouhin

kouhin commented Aug 7, 2016

We are using sensu to monitoring the server.
After upgrading from 1.11 to 1.12, we got alert from sensu like this

Check failed to run: statvfs() function failed: Permission denied, ["/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sys-filesystem-1.1.4/lib/unix/sys/filesystem.rb:201:in `stat'", "/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugins-disk-checks-1.0.2/bin/check-disk-usage.rb:99:in `check_mount'", "/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugins-disk-checks-1.0.2/bin/check-disk-usage.rb:94:in `block in fs_mounts'", "/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugins-disk-checks-1.0.2/bin/check-disk-usage.rb:86:in `each'", "/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugins-disk-checks-1.0.2/bin/check-disk-usage.rb:86:in `fs_mounts'", "/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugins-disk-checks-1.0.2/bin/check-disk-usage.rb:137:in `run'", "/opt/sensu/embedded/lib/ruby/gems/2.0.0/gems/sensu-plugin-1.2.0/lib/sensu-plugin/cli.rb:56:in `block in <class:CLI>'"] 

We tried to run df -h, this is the output

$ df -h
df: ‘/var/lib/docker/devicemapper/mnt/06ad84ddfd6fc597ea6c4da96ab9c0645e282c80a090fdf424aef7030adb3d45’: Permission denied
df: ‘/var/lib/docker/containers/c9f9f6223f6cad4411bd047db2ab6b5a9665731e8b62b65e0c5c78d9a1f17739/shm’: Permission denied
df: ‘/var/lib/docker/devicemapper/mnt/04ae9cebd44fc95159eaf7b378290e81bb829aebbc5de758a6a92c3a15b5a08a’: Permission denied
df: ‘/var/lib/docker/containers/bb7335cfdbd8af3ef45e3880ce2e6350b697a789607a2617b85a32f5fc944e52/shm’: Permission denied
df: ‘/var/lib/docker/devicemapper/mnt/4cb3f3aecde81cdac3f5b5c38acf3507abc89408d2e3e79e5c1bb320c0b144f8’: Permission denied
df: ‘/var/lib/docker/containers/3ccb534a4042436d7c54b1a9d8fece5de9c6cd99c465096c241b5a8b9f419358/shm’: Permission denied
df: ‘/var/lib/docker/devicemapper/mnt/2de82db393af917fd7b3e4171b44dbd109a89ded751bfbf12d3ac3906469be1f’: Permission denied
df: ‘/var/lib/docker/containers/80f0385a28946a3c58c7c0ef469a903924dd439252a98b2b3dfde5ea2f4c921c/shm’: Permission denied
df: ‘/var/lib/docker/devicemapper/mnt/9b5db9370ecfbea1e77dbc10c1da77fdc031fdc1d81a2a2f14da6820f76e0bca’: Permission denied
df: ‘/var/lib/docker/containers/483950f5a27fa574d2d98a4fcb51f657061fbb2ccb8b496b5f68c4d95a77d60b/shm’: Permission denied
df: ‘/var/lib/docker/devicemapper/mnt/94f591cb8b6a7da42675bf911817a78b35ac842ee5e99ddec07787f3670d9935’: Permission denied
df: ‘/var/lib/docker/containers/89bd4537e9bfff65f126e0cc190f930d1d5d9fbb6d01363e17112f6f01623f07/shm’: Permission denied
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       169G  5.0G  164G   3% /
devtmpfs         11G     0   11G   0% /dev
tmpfs            11G     0   11G   0% /dev/shm
tmpfs            11G  808K   11G   1% /run
tmpfs            11G     0   11G   0% /sys/fs/cgroup
/dev/vda1       497M  136M  362M  28% /boot
tmpfs           2.1G     0  2.1G   0% /run/user/22568

But this is the output when running 1.11

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       169G  4.2G  165G   3% /
devtmpfs         11G     0   11G   0% /dev
tmpfs            11G     0   11G   0% /dev/shm
tmpfs            11G  1.4M   11G   1% /run
tmpfs            11G     0   11G   0% /sys/fs/cgroup
/dev/vda1       497M  136M  362M  28% /boot
tmpfs           2.1G     0  2.1G   0% /run/user/22568

Then we found there are so many proc, net, shm in /etc/mtab after upgrading to 1.12,

/etc/mtab in 1.12

$ cat /etc/mtab
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=10705584k,nr_inodes=2676396,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/vda3 / xfs rw,relatime,attr2,inode64,noquota 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
/dev/vda1 /boot xfs rw,relatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/vda3 /var/lib/docker/devicemapper xfs rw,relatime,attr2,inode64,noquota 0 0
/dev/mapper/docker-253:3-301990042-06ad84ddfd6fc597ea6c4da96ab9c0645e282c80a090fdf424aef7030adb3d45 /var/lib/docker/devicemapper/mnt/06ad84ddfd6fc597ea6c4da96ab9c0645e282c80a090fdf424aef7030adb3d45 xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
proc net:[4026531956] proc rw,nosuid,nodev,noexec,relatime 0 0
shm /var/lib/docker/containers/c9f9f6223f6cad4411bd047db2ab6b5a9665731e8b62b65e0c5c78d9a1f17739/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/mapper/docker-253:3-301990042-04ae9cebd44fc95159eaf7b378290e81bb829aebbc5de758a6a92c3a15b5a08a /var/lib/docker/devicemapper/mnt/04ae9cebd44fc95159eaf7b378290e81bb829aebbc5de758a6a92c3a15b5a08a xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
shm /var/lib/docker/containers/bb7335cfdbd8af3ef45e3880ce2e6350b697a789607a2617b85a32f5fc944e52/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
/dev/mapper/docker-253:3-301990042-4cb3f3aecde81cdac3f5b5c38acf3507abc89408d2e3e79e5c1bb320c0b144f8 /var/lib/docker/devicemapper/mnt/4cb3f3aecde81cdac3f5b5c38acf3507abc89408d2e3e79e5c1bb320c0b144f8 xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
shm /var/lib/docker/containers/3ccb534a4042436d7c54b1a9d8fece5de9c6cd99c465096c241b5a8b9f419358/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
proc net:[4026532309] proc rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/docker-253:3-301990042-2de82db393af917fd7b3e4171b44dbd109a89ded751bfbf12d3ac3906469be1f /var/lib/docker/devicemapper/mnt/2de82db393af917fd7b3e4171b44dbd109a89ded751bfbf12d3ac3906469be1f xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
shm /var/lib/docker/containers/80f0385a28946a3c58c7c0ef469a903924dd439252a98b2b3dfde5ea2f4c921c/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
proc net:[4026532374] proc rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/docker-253:3-301990042-9b5db9370ecfbea1e77dbc10c1da77fdc031fdc1d81a2a2f14da6820f76e0bca /var/lib/docker/devicemapper/mnt/9b5db9370ecfbea1e77dbc10c1da77fdc031fdc1d81a2a2f14da6820f76e0bca xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
shm /var/lib/docker/containers/483950f5a27fa574d2d98a4fcb51f657061fbb2ccb8b496b5f68c4d95a77d60b/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
proc net:[4026532439] proc rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/docker-253:3-301990042-94f591cb8b6a7da42675bf911817a78b35ac842ee5e99ddec07787f3670d9935 /var/lib/docker/devicemapper/mnt/94f591cb8b6a7da42675bf911817a78b35ac842ee5e99ddec07787f3670d9935 xfs rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
shm /var/lib/docker/containers/89bd4537e9bfff65f126e0cc190f930d1d5d9fbb6d01363e17112f6f01623f07/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
proc net:[4026532506] proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/user/22568 tmpfs rw,nosuid,nodev,relatime,size=2142820k,mode=700,uid=22568,gid=10000 0 0

in 1.11

$ cat /etc/mtab
rootfs / rootfs rw 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=10705584k,nr_inodes=2676396,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls cgroup rw,nosuid,nodev,noexec,relatime,net_cls 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/vda3 / xfs rw,relatime,attr2,inode64,noquota 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
/dev/vda1 /boot xfs rw,relatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
tmpfs /run/user/22568 tmpfs rw,nosuid,nodev,relatime,size=2142820k,mode=700,uid=22568,gid=10000 0 0

I have google such as docker 1.12 /etc/mtab docker 1.12 volume, but found nothing.

Of course the alert can be fixed just by change the permission of sensu or docker daemon, but I want to know which is expected one ? Is there any issue or document about it , thank you!

Output of docker version:

$ docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        
 OS/Arch:      linux/amd64
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

BTW, I found the version of docker-containerd in 1.12 is

docker-containerd -v
containerd version 0.2.0 commit: 0ac3cd1be170d180b2baed755e8f0da547ceb267

while this is in 1.11

$ docker-containerd -v
containerd version 0.2.2 commit: 9dc2b3273db42c75368988a3885a3afd770069d9

Output of docker info:

(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.):

OpenStack

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Aug 8, 2016

Contributor

What host OS and init system?

EDIT:

Can you try setting MountFlags in your systemd unit file to "private" and then restart docker?

Contributor

cpuguy83 commented Aug 8, 2016

What host OS and init system?

EDIT:

Can you try setting MountFlags in your systemd unit file to "private" and then restart docker?

@kouhin

This comment has been minimized.

Show comment
Hide comment
@kouhin

kouhin Aug 9, 2016

@cpuguy83 Thank you!!!

After adding MountFlags=private or MountFlags=slave to /usr/lib/systemd/system/docker.service, mounted volumes disappeared.
I think it is caused by #22806

kouhin commented Aug 9, 2016

@cpuguy83 Thank you!!!

After adding MountFlags=private or MountFlags=slave to /usr/lib/systemd/system/docker.service, mounted volumes disappeared.
I think it is caused by #22806

@kouhin kouhin closed this Aug 9, 2016

@kouhin

This comment has been minimized.

Show comment
Hide comment
@kouhin

kouhin Aug 9, 2016

@cpuguy83 I also tried to change MountFlags=shared, it also works and mounted volumes are not listed in df -h.
Then I tried to find out default MountFlags value via systemctl show docker.

This is the result:

docker.service file systemctl show docker
No MountFlags MountFlags=0
MountFlags=shared MountFlags=1048576
MountFlags=slave MountFlags=524288
MountFlags=private MountFlags=262144

kouhin commented Aug 9, 2016

@cpuguy83 I also tried to change MountFlags=shared, it also works and mounted volumes are not listed in df -h.
Then I tried to find out default MountFlags value via systemctl show docker.

This is the result:

docker.service file systemctl show docker
No MountFlags MountFlags=0
MountFlags=shared MountFlags=1048576
MountFlags=slave MountFlags=524288
MountFlags=private MountFlags=262144
@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Aug 9, 2016

Contributor

The default is (supposed to be) shared, so I'm not sure why it would be any different.

Contributor

cpuguy83 commented Aug 9, 2016

The default is (supposed to be) shared, so I'm not sure why it would be any different.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment