volumes mounted in incorrect order

[Supermathie]

Looks like #5489 has struck again, but this time I think I've figured it out.

From inside the container I can't see one of the mountpoints:

root@tiefighter5-toontown:/# ls -al /shared/backups
ls: cannot access /shared/backups: No such file or directory

Inspecting the container in question shows they SHOULD be there:

    "Volumes": {
        "/shared": "/var/docker/shared/toontown",
        "/shared/backups": "/gv/backups/toontown",
        "/shared/uploads": "/gv/uploads/toontown",
        "/var/log": "/var/docker/shared/toontown/log/var-log"
    },

But checking /proc/mounts from inside the container (which is presumably correct) shows:

root@tiefighter5-toontown:/# grep shared /proc/mounts
tiefighter4:/gv /shared/backups fuse.glusterfs (...)
/dev/mapper/vg_sys-var /shared ext4  (...)
tiefighter4:/gv /shared/uploads fuse.glusterfs  (...)

It looks as though docker mounted /shared/backups first, then hid it with the /shared mount.

[Supermathie]

Restarting the container made it worse:

tiefighter4:/gv on /shared/uploads type fuse.glusterfs (...)
tiefighter4:/gv on /shared/backups type fuse.glusterfs (...)
/dev/mapper/vg_sys-var on /shared type ext4 (...)

Now I can't see anything in /shared.

[johnae]

Yes I've had, probably, the same issue on docker 1.2.0. Seems bind mounted volumes aren't always mounted. Actually it's sort of not mounted 70% of the time or something so it's not rare.

[cpuguy83]

There was some re-factoring that happened here, though I cannot reproduce this.

[cpuguy83]

Also, please post your docker info output. Thanks.

[johnae]

Here's my info:

Containers: 2
Images: 18
Storage Driver: devicemapper
 Pool Name: docker-202:1-524946-pool
 Pool Blocksize: 64 Kb
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 1256.4 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 1.9 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.13.0-34-generic
Operating System: Ubuntu 14.04.1 LTS
WARNING: No swap limit support

[cpuguy83]

@johnae And docker version?

[johnae]

@cpuguy83 as I think I mentioned it's 1.2.0. More specifically it's Docker version 1.2.0, build fa7b24f

[Supermathie]

@johnae The volumes probably are mounted, but the mountpoint is being hidden by another closer to the root.

What's the output of docker inspect container? What's in /proc/mounts in the container?

[johnae]

@Supermathie you may be right. I was using a third party container which was the one I was having trouble with. The instructions told me " to store data on the host, do this: -v /path/on/host/to/data/db:/data/db", however the container (Dockerfile) defines a volume at /data. I think that's what is causing the problems. Most of the time it wouldn't create the host mounted one or it was, as you say, obscured.

So /data is mounted via Dockerfile and that seems to already contain a db directory. Then I mount /some/path/here:/data/db and it gets obscured (about 70% of the time which is weird).

[Supermathie]

@cpuguy83 here you are:

$ docker info 
Containers: 21
Images: 62
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 108
Execution Driver: native-0.2
Kernel Version: 3.13.0-34-generic
Operating System: Ubuntu 14.04.1 LTS
WARNING: No swap limit support

$ docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

[cpuguy83]

@Supermathie Is this just an issue with /proc/mounts or is your stuff not actually being mounted?

[Supermathie]

@cpuguy83 Neither, really.

Technically the volumes are mounted but some of them are inaccessible due to being mounted in the incorrect order.

/proc/mounts is just showing me why my data is not available.

[Supermathie]

@cpuguy83 Have I sufficiently explained the problem?

[borsboom]

We are seeing this unpredictable volume mounting order as well. With 1.1.2, we could count on them being mounted in order, but with 1.2.0 it seems to be random. If you need an additional example to reproduce, I would be happy to provide one.

[crosbymichael]

I think this is because of the map randomization in go ~1.3

[tianon]

So we need something that can sort them in order from shallow to deep so
that we mount /a before /a/b ?

[Supermathie]

There's essentially two choices:

• keep original order
• sort from shallow to deep path depth

Sorting from shallow to deep probably makes the most sense. It'd be a fairly pathological case where keeping the original order was necessary and sorting from shallow to deep broke it.

[crosbymichael]

I think sorting is our best option because we cannot keep the order. Go randomizes maps and there is nothing we can do about that unless we create our own type with order information.

I think sorting is the best

[cpuguy83]

SGTM

[cpuguy83]

See #8055