New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with AutoFS mounts and docker 1.11.2 #24303

Closed
mogthesprog opened this Issue Jul 4, 2016 · 13 comments

Comments

Projects
None yet
5 participants
@mogthesprog

mogthesprog commented Jul 4, 2016

Have issues with bind mounts of nfs volumes inside containers, and on the host. The mounts work fine when no container is present. If the bind mount into the container is successful then it seems to break the mounts on the host. Note: The mounts are controlled by autofs. More details below.

Output of docker version:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:23:11 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:23:11 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: vg_root-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 15.24 GB
 Data Space Total: 107.4 GB
 Data Space Available: 92.13 GB
 Metadata Space Used: 2.486 MB
 Metadata Space Total: 33.55 MB
 Metadata Space Available: 31.07 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 251.7 GiB
Name: gb-doc-svb-0311.dunnhumby.co.uk
ID: GN45:FBRT:F7EV:ZZR4:IAVJ:2RPG:3RMR:QP6N:2NUG:EAPR:6JGT:Q6ZU
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

  • Baremetal docker instance
  • autofs mounted file systems on the host are bindmounted into the container :rw.

Steps to reproduce the issue:

  1. have autofs mounted files systems nfs3/4
  2. docker run -d -v /nfs/foo/bar:/nfs/foo/bar imagename
  3. on the host attempt to cd /nfs/foo/bar and you get the following error:

Describe the results you received:

[username@gb-doc-svb-0311 ~]$ cd /nfs/foo/bar
-bash: cd: /nfs/foo/bar: Too many levels of symbolic links

Describe the results you expected:

This has worked in previous versions of docker (client and server identical version numbers in all cases), including:

  • 1.9.1
  • 1.10.3
  • 1.7.1
  • 1.8.2

Additional information you deem important (e.g. issue happens only occasionally):

I'm still trying to narrow down the differences, but sometimes the mount errors are returned in the container, and sometimes their returned on the host. I thought that it may be related to the mountflags line in my unit file

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/docker daemon -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

[Install]
WantedBy=multi-user.target

I'm going to try and reproduce with MountFlags=shared and i'll see how i get on.

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Jul 4, 2016

Contributor

That seems like the most likely change that could well interact with autofs, let us know if that helps.

Contributor

justincormack commented Jul 4, 2016

That seems like the most likely change that could well interact with autofs, let us know if that helps.

@mogthesprog

This comment has been minimized.

Show comment
Hide comment
@mogthesprog

mogthesprog Jul 4, 2016

So interestingly... even with MountFlags=slave, the issue seems to go away when I bumped the autofs version number.

Had the idea after reading this post about bind mounting autofs mounts elsewhere in the filesystem. Seems to have resolved the issue. I'm gonna roll back. I'm going to try again with the older version of autofs to confirm that that's where the issue was coming from. post back in a bit.

Just to be clear, the working problematic autofs version was:

Name        : autofs
Arch        : x86_64
Epoch       : 1
Version     : 5.0.7
Release     : 48.el7

and the working version:

Name        : autofs
Arch        : x86_64
Epoch       : 1
Version     : 5.0.7
Release     : 54.el7

mogthesprog commented Jul 4, 2016

So interestingly... even with MountFlags=slave, the issue seems to go away when I bumped the autofs version number.

Had the idea after reading this post about bind mounting autofs mounts elsewhere in the filesystem. Seems to have resolved the issue. I'm gonna roll back. I'm going to try again with the older version of autofs to confirm that that's where the issue was coming from. post back in a bit.

Just to be clear, the working problematic autofs version was:

Name        : autofs
Arch        : x86_64
Epoch       : 1
Version     : 5.0.7
Release     : 48.el7

and the working version:

Name        : autofs
Arch        : x86_64
Epoch       : 1
Version     : 5.0.7
Release     : 54.el7
@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Jul 4, 2016

Contributor

Ah that post is interesting. I wonder what autofs is doing to work around this?

Contributor

justincormack commented Jul 4, 2016

Ah that post is interesting. I wonder what autofs is doing to work around this?

@mogthesprog

This comment has been minimized.

Show comment
Hide comment
@mogthesprog

mogthesprog Jul 4, 2016

okay.. so I was wrong. bumping the nfs version doesn't fix the problem. I'll go into more depth about how to recreate the problem....

So we have some autofs nfs mounts, which are mounted under nfs. If we create a container, which bind mounts the nfs folder -v /nfs:/nfs, and then proceed to ls the directory on the host then everything is fine. If i then docker exec into the container and do an ls, again, everything is fine.

Now, if i leave the container, umount the drive on the underlying host, and then attempt to remount it, by performing an ls of the nfs mount, then i get the following error:

[root@gb-wat-svv-1459 ~]# cd /nfs/foo
bash: cd: /nfs/foo: Too many levels of symbolic links

Note: I've tried this with MountFlags=shared and it doesn't resolve the issue.

This is with the latest stable version of autofs from the CentOS repository. and the docker version in the top comment of this post. Any guidance on where to look next? This wasn't an issue in previous versions of docker, i've just rolled back to 1.9.1, 1.10.3, 1.8.2, all of them work with the procedure above. It seems that only 1.11.2 is having issues.

Any ideas?

mogthesprog commented Jul 4, 2016

okay.. so I was wrong. bumping the nfs version doesn't fix the problem. I'll go into more depth about how to recreate the problem....

So we have some autofs nfs mounts, which are mounted under nfs. If we create a container, which bind mounts the nfs folder -v /nfs:/nfs, and then proceed to ls the directory on the host then everything is fine. If i then docker exec into the container and do an ls, again, everything is fine.

Now, if i leave the container, umount the drive on the underlying host, and then attempt to remount it, by performing an ls of the nfs mount, then i get the following error:

[root@gb-wat-svv-1459 ~]# cd /nfs/foo
bash: cd: /nfs/foo: Too many levels of symbolic links

Note: I've tried this with MountFlags=shared and it doesn't resolve the issue.

This is with the latest stable version of autofs from the CentOS repository. and the docker version in the top comment of this post. Any guidance on where to look next? This wasn't an issue in previous versions of docker, i've just rolled back to 1.9.1, 1.10.3, 1.8.2, all of them work with the procedure above. It seems that only 1.11.2 is having issues.

Any ideas?

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Jul 4, 2016

Contributor

Does it make a difference if you use -v /nfs:/nfs:shared?

Contributor

justincormack commented Jul 4, 2016

Does it make a difference if you use -v /nfs:/nfs:shared?

@mogthesprog

This comment has been minimized.

Show comment
Hide comment
@mogthesprog

mogthesprog Jul 4, 2016

That seems to have fixed it! Thank you!

Fancy explaining what :shared does? Couldn't see it in the docs...

again, cheers

mogthesprog commented Jul 4, 2016

That seems to have fixed it! Thank you!

Fancy explaining what :shared does? Couldn't see it in the docs...

again, cheers

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Jul 4, 2016

Contributor

The :shared means that the /nfs mount shares mount changes with the host, which for autofs is what you want, but it is not the default, which is to have a private copy of the mount, which seems to cause issues, probably related to the ones in the linked kernel issues.

Contributor

justincormack commented Jul 4, 2016

The :shared means that the /nfs mount shares mount changes with the host, which for autofs is what you want, but it is not the default, which is to have a private copy of the mount, which seems to cause issues, probably related to the ones in the linked kernel issues.

@mogthesprog

This comment has been minimized.

Show comment
Hide comment
@mogthesprog

mogthesprog Jul 5, 2016

Thanks for the info. Makes sense. I'm guessing this :shared is similar to the MountFlags in the docker daemon unit file in some way?

mogthesprog commented Jul 5, 2016

Thanks for the info. Makes sense. I'm guessing this :shared is similar to the MountFlags in the docker daemon unit file in some way?

@mogthesprog mogthesprog closed this Jul 5, 2016

@mogthesprog

This comment has been minimized.

Show comment
Hide comment
@mogthesprog

mogthesprog Jul 11, 2016

Just to clarify, the fix was to add the following to the service file

MountFlags=shared

and then when running the container add :shared to the volume mount.

mogthesprog commented Jul 11, 2016

Just to clarify, the fix was to add the following to the service file

MountFlags=shared

and then when running the container add :shared to the volume mount.

@chrert

This comment has been minimized.

Show comment
Hide comment
@chrert

chrert Aug 30, 2016

I'm facing the same problem right now. The difference is that I'm using docker-compose. I tried to add :shared to the volume mount, but it gives an error:
invalid bind mount spec "test_mount_test:/opt/test:shared"

Does anyone have a suggestion?

chrert commented Aug 30, 2016

I'm facing the same problem right now. The difference is that I'm using docker-compose. I tried to add :shared to the volume mount, but it gives an error:
invalid bind mount spec "test_mount_test:/opt/test:shared"

Does anyone have a suggestion?

@justincormack

This comment has been minimized.

Show comment
Hide comment
@justincormack

justincormack Aug 30, 2016

Contributor

That may well be an error from compose, perhaps it does not support shared
yet? You could open an issue on compose for this.

On 30 Aug 2016 11:04 a.m., "chrert" notifications@github.com wrote:

I'm facing the same problem right now. The difference is that I'm using
docker-compose. I tried to add :shared to the volume mount, but it gives
an error:
invalid bind mount spec "test_mount_test:/opt/test:shared"

Does anyone have a suggestion?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#24303 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPD3iBVflIqzK_3pkiRQt8_q0rHaGks5qlAA-gaJpZM4JENnn
.

Contributor

justincormack commented Aug 30, 2016

That may well be an error from compose, perhaps it does not support shared
yet? You could open an issue on compose for this.

On 30 Aug 2016 11:04 a.m., "chrert" notifications@github.com wrote:

I'm facing the same problem right now. The difference is that I'm using
docker-compose. I tried to add :shared to the volume mount, but it gives
an error:
invalid bind mount spec "test_mount_test:/opt/test:shared"

Does anyone have a suggestion?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#24303 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPD3iBVflIqzK_3pkiRQt8_q0rHaGks5qlAA-gaJpZM4JENnn
.

@chrert

This comment has been minimized.

Show comment
Hide comment
@chrert

chrert Aug 30, 2016

Ok thanks! I will open an issue for that...

Btw.: Is this feature documented somewhere?

chrert commented Aug 30, 2016

Ok thanks! I will open an issue for that...

Btw.: Is this feature documented somewhere?

spacemansteve pushed a commit to spacemansteve/ADSimportpipeline that referenced this issue Dec 6, 2016

SpacemanSteve SpacemanSteve
adsabs#119 support for new disk mounts
the earlier attempt at hard mounts did not work, paths are hard coded in site packages
so we are trying to use shared mounts in Docker as suggested by moby/moby#24303
@SteveHarris

This comment has been minimized.

Show comment
Hide comment
@SteveHarris

SteveHarris Feb 1, 2017

Thank you, I was having the same problem and this is fixed in my version of docker-compose - I added the ":shared" and now the autofs mounts seem to be working.

SteveHarris commented Feb 1, 2017

Thank you, I was having the same problem and this is fixed in my version of docker-compose - I added the ":shared" and now the autofs mounts seem to be working.

spacemansteve pushed a commit to spacemansteve/ADSimportpipeline that referenced this issue May 31, 2017

SpacemanSteve SpacemanSteve
adsabs#119 support for new disk mounts
the earlier attempt at hard mounts did not work, paths are hard coded in site packages
so we are trying to use shared mounts in Docker as suggested by moby/moby#24303
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment