Docker-compose and incorrect absolute paths for volumes

[ej3]

windows 10 build 1503.11 running Xenial 16.04.2 LTS with docker client 17.03.0 connecting to the hyper-V docker-for-windows daemon 17.03.1 and docker-compose 1.11.1

From within wsl things like docker run --rm -v c:/my/stuff:/var/goes/here bash ls /var/goes/here work fantastically. This command will simply list the contents of c:/my/stuff.

What does not work is If you have a docker-compose.yml that looks like:

version: '2'
  services:
    bash_it:
      image: bash
      volumes:
        - ./:/var/goes/here

and you run something like docker-compose -p test -d bash_it up && docker exec test_bash_it_1 ls /var/goes/here. Everything will explode and die - you may have to restart the docker-for-windows daemon / engine.

the error will look something like:

ERROR: for proxy  Cannot start service proxy: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"ro
otfs_linux.go:54: mounting \\\"/mnt/c/my/stuff\\\" to rootfs \\\"/var/lib/docker/overlay2/e4bda146bae2ce38c13ad8f3c4ff5f6e
6f3198041ce02d4c6f593f65948e7086/merged\\\" at \\\"/var/lib/docker/overlay2/e4bda146bae2ce38c13ad8f3c4ff5f6e6f3198041ce02d4c6f593f65948e7086/merged/var/goes/here\\\"
 caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.

It's all rather complicated looking, but I'm pretty sure I can tell you what's going on: the windows-for-docker daemon expects a valid windows path (eg c:/my/stuff) BUT docker-compose is calling os.path.abspath to resolve the absolute path of the resource and it's getting a URI/path that looks like /mnt/c/my/stuff - which is absolute nonsense unless you're in the windows linux subsytem.. which the hyper-v daemon is clearly not.

so... that's an interesting problem. Is there a switch like MSYS_NO_PATHCONV=1 or something I can do to prevent os.path.abspath from returning a wsl alias? OR - how should this work?

[aseering]

So, this is a little more conceptually complicated: Unlile in Cygwin, /mnt/c/... is not an alias in WSL. It is the canonical and only way to refer to that directory. C:\... is not a valid Linux path; therefore not a valid WSL path. (C:\ would immediately cause errors on Linux, so WSL does likewise.)

In this particular case, maybe you could try putting an absolute Windows path in your config file? You might also want to bring this up on the Docker forums -- they seem to be pretty active, and the Docker team seems to care a lot about compatibility. (And this would be a really easy, if kind of hackish, thing for them to work around.)

[sakai135]

os.path.abspath returning /mnt/c/path within WSL makes total sense though. The issue is the mismatch between /mnt/c of WSL and /c in the VM of Docker for Windows.

Within the Hyper-V VM, the C: is mounted to /c instead of /mnt/c. I believe docker/for-win#371 would be the ideal solution, which is to make Docker for Windows mount C: into /mnt/c inside the VM.

[therealkenc]

Within the Hyper-V VM, the C: is mounted to /c instead of /mnt/c

And in MSYS as well, which is how the whole Windows Git BASH world sees things. I know that mounting remote shares on / is an anathema on Real Linux, which WSL strives to follow. But there is (unfortunately) a lot of precedent for that convention, and WSL goes against the grain here. I imagine this will sort itself once WSL supports mounting of DrvFS on arbitrary paths.

[aseering]

Keiichi Shimamura (@sakai135) -- ah, interesting: I initially assumed Docker for Windows did Windows path translation server-side but it would make a lot of sense for it to do it client-side (in which case the Linux client would indeed need to pass in a VM-compatible path).

therealkenc -- ooh, that's a good thought: WSL already supports bind-mounts. Therefore it does (in essence) support mounting DrvFS on arbitrary paths. So, proposed workaround (untested):

$ sudo mkdir /c
$ sudo mount --bind /mnt/c /c
$ cd /c/path/to/project
$ docker-compose ...

[sakai135]

Adam Seering (@aseering) the bind mount worked for me :) Thanks for the workaround!

[therealkenc]

I forgot binding works. Perfect. Once real init (aka systemd) is working, I am imagining that line being in everyone's startup script....

[ej3]

Adam Seering (@aseering) you made me realize that it was unusual that the WSL docker client was taking non-canonical paths as arguments, so I went back and tried

$ docker run --rm -v /mnt/c/Users/ej3/dev/play/:/var/play bash ls -la /var/play
total 4
drwxr-xr-x    5 root     root           100 Apr  5 21:05 .
drwxr-xr-x    1 root     root          4096 Apr  5 21:47 ..
drwxr-xr-x    2 root     root            40 Apr  5 20:58 Dockerfile
drwxr-xr-x    2 root     root            40 Apr  5 21:02 configure.file
drwxr-xr-x    2 root     root            40 Apr  5 21:05 test.dev.conf

which seemed to work! (but the listing wasn't complete?!) so then I had no idea what the problem could have been, and I panicked a bit.

Then the conversation went on, and the bind for DrvFS came up:

$ docker run --rm -v /c/Users/ej3/dev/play/:/var/play bash ls -la /var/play
total 12
drwxr-xr-x    2 root     root          4096 Apr  5 21:44 .
drwxr-xr-x    1 root     root          4096 Apr  5 21:47 ..
drwxr-xr-x    2 root     root             0 Apr  5 21:44 .git
-rwxr-xr-x    1 root     root            81 Apr  5 21:10 Dockerfile
-rwxr-xr-x    1 root     root           675 Apr  5 21:44 README.md
-rwxr-xr-x    1 root     root           139 Apr  5 21:19 bash_it.sh
-rwxr-xr-x    1 root     root           116 Apr  5 21:04 docker-compose.yml
-rwxr-xr-x    1 root     root            39 Apr  5 21:09 test.conf
-rwxr-xr-x    1 root     root            40 Apr  5 21:08 test.dev.conf
$

Then I was able to reproduce my issue with a simple docker-compose example, and the bind completely resolves the issue.

Just a further note: When I first tried using the bind I was getting a new set of IO errors because I had created the containers with docker-compose on the /mnt/c volume, moved to the /c volume and simply did a docker-compose up without recreating the containers from the images. If anyone else has this problem: be sure to re-instantiate new containers from the original images after you switch mount points.

Thanks.

[teoulas]

I wanted to do the same a few months ago. I ended up simply symlinking /mnt/c to /c:

sudo ln - s /mnt/c /c

Not sure if it's better performance-wise than bind-mounting, but it works just as well.

[aseering]

For what it's worth, there are some applications (Web servers in particular) that will deliberately refuse to read files from any path that contains a symlink. Usually for security reasons, because malicious users could use symlinks to escape DocumentRoot. Bind mounts usually don't suffer from this limitation -- anyone can create a symlink anywhere, but you need elevated permissions (traditionally, root) to set up a bind-mount, so sysadmins can still use them but they're harder to exploit.

Hi folks,

We're talking this over internally at Microsoft and open to suggestions.

From Russ on the WSL (Windows Subsystem for Linux) team: "Bind and symlinks both work, but the bind may be better in some cases since not all services will follow symlinks. Downside of bind is that it will not stick between WSL instances due to our init."

Essentially agreeing with Adam Seering (@aseering)'s points :)

What ideas do you have? What are you trying to do when you run into problems around file paths?

Cheers,
Myles

[tsafin]

From my holistic point of view, the ideal solution would be for all us (and I really expect it happen some day), iff WSL will converge with Docker and Docker on Windows will move to use WSL compatibility layer instead of MobyLinux VM. In this case Docker Compose could be used the way it is used on Linux and transparently rely on WSL underneath.

Thanks for the suggestions!