Share gluster volume though volume-from

[peijunfu]

In our using scenarios we expect that there is one container in host which management of the mount different FS e.g. GlusterFS. All other application container can volume-form that container to using the Gluster volumes.
But it seems current docker did not support this. For example: a container which installed Gluster client and fuse can successfully mount the Gluster volumes. But that mount volumes can't share with host or share with other containers on that host.

There are some solutions:

1.A Docker host mount a Gluster filesystem whithout Gluster client package, Gluster client are in one container. But it seems not working since one parameter "-v /proc:/proc" failed.
https://github.com/rootfs/install-glusterfs-on-fc21

2.Docker volume plugin for GlusterFS. But it is required that Gluster client are installed on host beforehead.
https://github.com/calavera/docker-volume-glusterfs

3.K8s also have example for using glusterfs. But similarly, Gluster client shall be installed on host beforehead.
http://kubernetes.io/v1.1/examples/glusterfs/README.html

In short, i expect that in host, it is not require to install Gluster client package, but in one container which mount the gluster volume that can be share to other container.

Any suggestion, comments, or more official solution?

[thaJeztah]

We're looking into making it easier to run volume plugins in a container, but don't have a full design for that yet. Currently, installing the requirements for glusterfs on the host, and using the glusterfs volume-plugin is the recommended approach, but ymmv

[phemmer]

a container which installed Gluster client and fuse can successfully mount the Gluster volumes. But that mount volumes can't share with host or share with other containers on that host.

Can you provide your docker version, and a reproduction example? This should be working.

[cpuguy83]

Mounts done in a container are not, by default, shared with the host, and as such do not work with --volumes-from.

In order to share mounts with the host you need to bind-mount a host dir and mark it as shared: -v /foo:/foo:shared, or use rshared instead of shared for recursive sharing beneath the main mountpoint.
In addition, /foo on the host would need to be marked as a shared mount (or the root mount on the host needs to be shared)... you can accomplish this like so: mount --bind /foo /foo && mount --make-shared /foo

[peijunfu]

Sorry for late respone. First thank you all for comments.

@cpuguy83

According to above understand i made a test on docker version: 1.10.2
Operation on docker host:

1. mkdir /mountpoint
2. mount --bind /mountpoint /mountpoint && mount --make-shared /mountpoint
3. docker run -it --privileged=true -h client -v /mountpoint:/mountpoint:shared gfs:latest /bin/bash

But i got bellow error
"docker: Error response from daemon: Cannot start container 632a78539ea6b32c344a15c85e57d67b4005f73f93b11f30fb0aa76323bd1931: Path /mountpoint is mounted on /mountpoint but it is not a shared mount.."

Then i found:
#17034

which mentioned above error and your solution, but it seems still can't work in my docker host.

1. mount --make-shared /
2. docker run -v /foo:/bar:shared busybox
   docker: Error response from daemon: Cannot start container 6a2382a366e84a2f54819c6e413fe113b12efbcd0a46c840fba7989a6524de5e: Path /foo is mounted on /foo but it is not a shared mount..

My docker host is RHEL7.1 3.10.0-327.4.4.el7.x86_64 (using root to execute command)
if i using findmnt -o TARGET,PROPAGATION /mountpoint, it seems also shared.
TARGET PROPAGATION
/mountpoint shared
/mountpoint shared
/mountpoint shared

Any comments? Thank you.

[peijunfu]

Does shared volume need other setting then bellow steps described? it seems still have problem of "not a shared mount". Could you give some suggestion?
Thank you.

Output of docker version:

[root@localhost /]# docker version
Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 16:16:33 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 16:16:33 2016
 OS/Arch:      linux/amd64

Output of docker info:

[root@localhost /]# docker info
Containers: 16
 Running: 0
 Paused: 0
 Stopped: 16
Images: 33
Server Version: 1.10.2
Storage Driver: devicemapper
 Pool Name: docker-253:0-25780433-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem:
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.475 GB
 Data Space Total: 107.4 GB
 Data Space Available: 4.375 GB
 Metadata Space Used: 3.142 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 489.4 MiB
Name: localhost.localdomain
ID: VOE3:6Y6O:BJAC:PTZK:JDJG:U4IW:7IVX:EN7J:3YU3:SDOB:ESIB:ILIQ
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Provide additional environment details (AWS, VirtualBox, physical, etc.):
VirtualBox and os info

[root@localhost /]# cat /etc/*release*
CentOS Linux release 7.2.1511 (Core)
Derived from Red Hat Enterprise Linux 7.2 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.2.1511 (Core)
CentOS Linux release 7.2.1511 (Core)
cpe:/o:centos:centos:7

List the steps to reproduce the issue:

1. mount --make-shared /
2. mount --bind foo foo
3. mount --make-shared foo
4. docker run -v /foo:/bar:shared busybox

Describe the results you received:

docker: Error response from daemon: Cannot start container ef3876618c5c013f695ee16560162f3385a7e5e8df0ea21d3636da4d9d34ee33: Path /foo is mounted on / but it is not a shared mount..

Describe the results you expected:

Provide additional info you think is important:

Mar 01 01:40:31 localhost.localdomain docker[10206]: time="2016-03-01T01:40:31.880837229-05:00" level=error msg="error locating sandbox id 7b437acc487056ceb5b7e4699c53582db77182e3250db7cd2d6260b391d048e8: sandbox 7b437acc487056ceb5b7e4699c53582db77182e3250db7cd2d6260b391d048e8 not found"
Mar 01 01:40:31 localhost.localdomain docker[10206]: time="2016-03-01T01:40:31.880886101-05:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/4a227e248f86575d3e27475c17c6cf10aa577309fb171c59b0ecc2339cdbb30a/shm: invalid argument"
Mar 01 01:40:31 localhost.localdomain docker[10206]: time="2016-03-01T01:40:31.880903691-05:00" level=error msg="Error unmounting container 4a227e248f86575d3e27475c17c6cf10aa577309fb171c59b0ecc2339cdbb30a: not mounted"
Mar 01 01:40:31 localhost.localdomain docker[10206]: time="2016-03-01T01:40:31.881000909-05:00" level=error msg="Handler for POST /v1.22/containers/4a227e248f86575d3e27475c17c6cf10aa577309fb171c59b0ecc2339cdbb30a/start returned error: Cannot start container 4a227e248f86575d3e27475c17c6cf10aa577309fb171c59b0ecc2339cdbb30a: Path /foo is mounted on /foo but it is not a shared mount."

[peijunfu]

I found the root case form this ticket, change the docker systemd "MountFlags=slave"
-> "MountFlags=share". Then it works. Is that so this changes is the offical way to do so?

#19625 (comment)

There are another ticket which proposal of removing MountFlags=slave from systemd unit definitions. Maybe in the furture, it may not need to manually change docker systemd unit.

docker/machine#3029

Anyway, the above solution met my requirement. That's very nice. Thank you all.

[rhvgoyal]

@peijunfu for shared to work, yes you will have to remove MountFlags=slave from systemd unit file and run docker daemon in host mount namespace. slave should work as it is.

I think @rhatdan has made changes to docker systemd unit file in fedora rawhide and removed the MountFlags=slave

[rhatdan]

Correct we have made those changes in Rawhide and will do so in rhel-7.2.4 release (docker-1.10)

[peijunfu]

@rhatdan @rhvgoyal Great, thank you for the information.