New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker-compose and incorrect absolute paths for volumes #1854

Closed
ej3 opened this Issue Apr 4, 2017 · 40 comments

Comments

Projects
None yet
@ej3

ej3 commented Apr 4, 2017

windows 10 build 1503.11 running Xenial 16.04.2 LTS with docker client 17.03.0 connecting to the hyper-V docker-for-windows daemon 17.03.1 and docker-compose 1.11.1

From within wsl things like docker run --rm -v c:/my/stuff:/var/goes/here bash ls /var/goes/here work fantastically. This command will simply list the contents of c:/my/stuff.

What does not work is If you have a docker-compose.yml that looks like:

version: '2'
  services:
    bash_it:
      image: bash
      volumes:
        - ./:/var/goes/here

and you run something like docker-compose -p test -d bash_it up && docker exec test_bash_it_1 ls /var/goes/here. Everything will explode and die - you may have to restart the docker-for-windows daemon / engine.

the error will look something like:

ERROR: for proxy  Cannot start service proxy: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"ro
otfs_linux.go:54: mounting \\\"/mnt/c/my/stuff\\\" to rootfs \\\"/var/lib/docker/overlay2/e4bda146bae2ce38c13ad8f3c4ff5f6e
6f3198041ce02d4c6f593f65948e7086/merged\\\" at \\\"/var/lib/docker/overlay2/e4bda146bae2ce38c13ad8f3c4ff5f6e6f3198041ce02d4c6f593f65948e7086/merged/var/goes/here\\\"
 caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.

It's all rather complicated looking, but I'm pretty sure I can tell you what's going on: the windows-for-docker daemon expects a valid windows path (eg c:/my/stuff) BUT docker-compose is calling os.path.abspath to resolve the absolute path of the resource and it's getting a URI/path that looks like /mnt/c/my/stuff - which is absolute nonsense unless you're in the windows linux subsytem.. which the hyper-v daemon is clearly not.

so... that's an interesting problem. Is there a switch like MSYS_NO_PATHCONV=1 or something I can do to prevent os.path.abspath from returning a wsl alias? OR - how should this work?

@aseering

This comment has been minimized.

Contributor

aseering commented Apr 4, 2017

So, this is a little more conceptually complicated: Unlile in Cygwin, /mnt/c/... is not an alias in WSL. It is the canonical and only way to refer to that directory. C:\... is not a valid Linux path; therefore not a valid WSL path. (C:\ would immediately cause errors on Linux, so WSL does likewise.)

In this particular case, maybe you could try putting an absolute Windows path in your config file? You might also want to bring this up on the Docker forums -- they seem to be pretty active, and the Docker team seems to care a lot about compatibility. (And this would be a really easy, if kind of hackish, thing for them to work around.)

@sakai135

This comment has been minimized.

sakai135 commented Apr 5, 2017

os.path.abspath returning /mnt/c/path within WSL makes total sense though. The issue is the mismatch between /mnt/c of WSL and /c in the VM of Docker for Windows.

Within the Hyper-V VM, the C: is mounted to /c instead of /mnt/c. I believe docker/for-win#371 would be the ideal solution, which is to make Docker for Windows mount C: into /mnt/c inside the VM.

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Apr 5, 2017

Within the Hyper-V VM, the C: is mounted to /c instead of /mnt/c

And in MSYS as well, which is how the whole Windows Git BASH world sees things. I know that mounting remote shares on / is an anathema on Real Linux, which WSL strives to follow. But there is (unfortunately) a lot of precedent for that convention, and WSL goes against the grain here. I imagine this will sort itself once WSL supports mounting of DrvFS on arbitrary paths.

@aseering

This comment has been minimized.

Contributor

aseering commented Apr 5, 2017

@sakai135 -- ah, interesting: I initially assumed Docker for Windows did Windows path translation server-side but it would make a lot of sense for it to do it client-side (in which case the Linux client would indeed need to pass in a VM-compatible path).

@therealkenc -- ooh, that's a good thought: WSL already supports bind-mounts. Therefore it does (in essence) support mounting DrvFS on arbitrary paths. So, proposed workaround (untested):

$ sudo mkdir /c
$ sudo mount --bind /mnt/c /c
$ cd /c/path/to/project
$ docker-compose ...
@sakai135

This comment has been minimized.

sakai135 commented Apr 5, 2017

@aseering the bind mount worked for me :) Thanks for the workaround!

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Apr 5, 2017

I forgot binding works. Perfect. Once real init (aka systemd) is working, I am imagining that line being in everyone's startup script....

@ej3

This comment has been minimized.

ej3 commented Apr 5, 2017

@aseering you made me realize that it was unusual that the WSL docker client was taking non-canonical paths as arguments, so I went back and tried

$ docker run --rm -v /mnt/c/Users/ej3/dev/play/:/var/play bash ls -la /var/play
total 4
drwxr-xr-x    5 root     root           100 Apr  5 21:05 .
drwxr-xr-x    1 root     root          4096 Apr  5 21:47 ..
drwxr-xr-x    2 root     root            40 Apr  5 20:58 Dockerfile
drwxr-xr-x    2 root     root            40 Apr  5 21:02 configure.file
drwxr-xr-x    2 root     root            40 Apr  5 21:05 test.dev.conf

which seemed to work! (but the listing wasn't complete?!) so then I had no idea what the problem could have been, and I panicked a bit.

Then the conversation went on, and the bind for DrvFS came up:

$ docker run --rm -v /c/Users/ej3/dev/play/:/var/play bash ls -la /var/play
total 12
drwxr-xr-x    2 root     root          4096 Apr  5 21:44 .
drwxr-xr-x    1 root     root          4096 Apr  5 21:47 ..
drwxr-xr-x    2 root     root             0 Apr  5 21:44 .git
-rwxr-xr-x    1 root     root            81 Apr  5 21:10 Dockerfile
-rwxr-xr-x    1 root     root           675 Apr  5 21:44 README.md
-rwxr-xr-x    1 root     root           139 Apr  5 21:19 bash_it.sh
-rwxr-xr-x    1 root     root           116 Apr  5 21:04 docker-compose.yml
-rwxr-xr-x    1 root     root            39 Apr  5 21:09 test.conf
-rwxr-xr-x    1 root     root            40 Apr  5 21:08 test.dev.conf
$

Then I was able to reproduce my issue with a simple docker-compose example, and the bind completely resolves the issue.

Just a further note: When I first tried using the bind I was getting a new set of IO errors because I had created the containers with docker-compose on the /mnt/c volume, moved to the /c volume and simply did a docker-compose up without recreating the containers from the images. If anyone else has this problem: be sure to re-instantiate new containers from the original images after you switch mount points.

Thanks.

@teoulas

This comment has been minimized.

teoulas commented Apr 11, 2017

I wanted to do the same a few months ago. I ended up simply symlinking /mnt/c to /c:

sudo ln - s /mnt/c /c

Not sure if it's better performance-wise than bind-mounting, but it works just as well.

@aseering

This comment has been minimized.

Contributor

aseering commented Apr 11, 2017

For what it's worth, there are some applications (Web servers in particular) that will deliberately refuse to read files from any path that contains a symlink. Usually for security reasons, because malicious users could use symlinks to escape DocumentRoot. Bind mounts usually don't suffer from this limitation -- anyone can create a symlink anywhere, but you need elevated permissions (traditionally, root) to set up a bind-mount, so sysadmins can still use them but they're harder to exploit.

@mylesbkeating1993

This comment has been minimized.

mylesbkeating1993 commented Apr 25, 2017

Hi folks,

We're talking this over internally at Microsoft and open to suggestions.

From Russ on the WSL (Windows Subsystem for Linux) team: "Bind and symlinks both work, but the bind may be better in some cases since not all services will follow symlinks. Downside of bind is that it will not stick between WSL instances due to our init."

Essentially agreeing with @aseering's points :)

What ideas do you have? What are you trying to do when you run into problems around file paths?

Cheers,
Myles

@tsafin

This comment has been minimized.

tsafin commented Apr 26, 2017

From my holistic point of view, the ideal solution would be for all us (and I really expect it happen some day), iff WSL will converge with Docker and Docker on Windows will move to use WSL compatibility layer instead of MobyLinux VM. In this case Docker Compose could be used the way it is used on Linux and transparently rely on WSL underneath.

@mylesbkeating1993

This comment has been minimized.

mylesbkeating1993 commented Apr 26, 2017

Thanks for the suggestions!

@edrevo

This comment has been minimized.

edrevo commented Jun 1, 2017

Do you need to sudo mount --bind /mnt/c /c everytime you open up a bash? I've tried adding the following line to /etc/fstab but that doesn't seem to do the trick:

C: /c drvfs rw,noatime 0 0
@DeadPixelz01

This comment has been minimized.

DeadPixelz01 commented Jun 7, 2017

Has anyone found a way to make sudo mount --bind /mnt/c /c permanent yet? I know I could just add it to something like ~/.zshrc .However, that requires me to enter a password every time I want to open the terminal >.>. It would be nice if we could add it to something like /etc/rc.local and have it automatically start up when I run the terminal

@therealkenc

This comment has been minimized.

Collaborator

therealkenc commented Jun 7, 2017

There's a long thread regarding daemons and startup tasks in #834, with a reasonable work-around at the very bottom. Short story, add mount to your /etc/sudoers so you don't have to enter the password every time. The User Voice is here.

const-g added a commit to const-g/docker-node-app that referenced this issue Aug 13, 2017

@copitz

This comment has been minimized.

copitz commented Aug 14, 2017

@reidblomquist

This comment has been minimized.

reidblomquist commented Nov 28, 2017

oh my god @aseering ily. you just helped me finally jump through the last of the many fiery hoops that line the pathway to getting docker fully operable from within lxss

@pd93

This comment has been minimized.

pd93 commented Jan 31, 2018

@aseering Not to pester, but is there any update on this? I'm struggling to get the bind mount workaround to work reliably. IMO, This is one of the last things holding back the WSL finally being a decent developer environment on Windows.

@aseering

This comment has been minimized.

Contributor

aseering commented Jan 31, 2018

Hm, I don't know of any update. But I'm not actually a WSL dev; just a user. Maybe someone on the WSL team can comment?

For the bind-mount, what have you tried? And have you tried a symlink instead? We might be able to help you with those, too.

@villasv

This comment has been minimized.

villasv commented Feb 1, 2018

Just as a reference for anyone facing this with Docker Toolbox (using VirtualBox by default).

Either make the bind or the symbolic link sugested above inside the docker-machine environment: sudo ln -s /c/ /mnt/c

@dihmuzikien

This comment has been minimized.

dihmuzikien commented Feb 3, 2018

Has anyone gotten their files corrupted by modifying files through mounted C drive to /c/ instead of /mnt/c? Specifically, I'm working on an issue in docker/composerepo. My.git/index` would get corrupted sometime after running the test scripts.

I tried both
sudo ln -s /c/ /mnt/c
and
sudo mount --bind /mnt/c /c

@pd93

This comment has been minimized.

pd93 commented Feb 8, 2018

@aseering Thanks for the offer of help. I did eventually get this working with the bind mount. Turned out my config was the issue - of course.

Today Microsoft released insiders build 17093. In the release notes it states:

WSL now processes the /etc/fstab file during instance start [GH 2636].

I thought this might be of interest to @edrevo who previously mentioned fstab was not working. Would this help us to keep the bind mount permanent rather than entering it into our .bashrc or equivalent?

The notes also mention:

AF_UNIX allows socket connections between Linux processes on WSL and Windows native processes

Would anyone with a bit more knowledge be able to say if these changes affect this issue at all?

@benhillis

This comment has been minimized.

Member

benhillis commented Feb 8, 2018

@pd93 - yes the addition of fstab support adds a better way to configure mounts. AF_UNIX support is new functionality that allows WSL processes and Windows processes to communicate and should not impact this. See the blog post for more details.

@aseering

This comment has been minimized.

Contributor

aseering commented Feb 8, 2018

Regarding AF_UNIX, in principle Docker for Windows should be able to use that to allow WSL/Linux applications to talk over a local socket (just like native Linux Docker) rather than TCP/loopback. I haven't read this part of Docker's source code, but I would guess that they would need to make some small modifications to their code: Previously, AF_UNIX didn't work and wasn't available on Windows, so existing applications will often avoid even trying it on Windows.

@villasv

This comment has been minimized.

villasv commented Feb 8, 2018

There is OTOH one new feature that might be of relevance here. On wsl.conf you can now specify the mount point of the C: drive:

Key: root Value: String value (/mnt/…)
Default: “/mnt/”
This key lets you define a new directory where fixed drives will be automatically mounted. For example, if you have a directory in WSL at “/windir/” and you specify that as the root, you would expect to see your fixed drives (for example, C:/) mounted at “/windir/c”

It also added other DrvFS options handling.

@pd93

This comment has been minimized.

pd93 commented Feb 8, 2018

@villasv I can confirm that adding the following to the /etc/wsl.conf file on 17093 appears to work:

[automount]
root = /

I brought up my docker project and can access all of my volume mounts 👍

@pd93

This comment has been minimized.

pd93 commented Feb 12, 2018

@benhillis Is there a way to obtain the root mount point from from the wsl command line without having to parse the wsl.conf file?

@benhillis

This comment has been minimized.

Member

benhillis commented Feb 12, 2018

@pd93 - No there is not.

@aseering

This comment has been minimized.

Contributor

aseering commented Feb 13, 2018

@pd93 you can get similar information by parsing /proc/mounts. Not clear that's any better :-) But it's more Linux-ey; existing Linux tools already support it.

@dwatrous

This comment has been minimized.

dwatrous commented Apr 14, 2018

I can get this to work, but it doesn't respect the file permissions I set in WSL. Thanks to recent changes, I can set keys on my windows system to 600 through WSL (I think it uses the drvfs filesystem and attaches some metadata to it). These changes aren't visible on the volume mount. Every file ends up with the same permissions, just like when I run docker from powershell.

@eiva

This comment has been minimized.

eiva commented Apr 25, 2018

How to access WSL filesystem from composer?
Each folder I ‘mount’ using volumes command appear empty in container...
The workarounds described here shows how to access windows drives, not WSL...

@kskalski

This comment has been minimized.

kskalski commented May 2, 2018

Did anybody experience mounted volumes to list just partial contents when accessed from container? I followed the hints provided here to mount bind /mnt/c as /c and /mnt/d as /d, as well as using symlinks, but the behavior of listing volume from docker container sometimes shows empty dir, sometimes only some of the files/directories that are present on host, sometimes all contents.

I'm using Docker Toolbox with docker machine on Virtual Box and command like
docker run --rm -v /d/rep/:/var/play bash ls -la /var/play
to test

@ej3

This comment has been minimized.

ej3 commented May 2, 2018

@kskalski I did before mounting the volume to the root. This could be happening because your mount --bind /mnt/d /d is not actually occurring / executing cleanly?

@kskalski

This comment has been minimized.

kskalski commented May 3, 2018

The mounts seems to be working fine, I can access all content without problem in /d, however when I mount /d as volume to docker container, it shows empty dirs (seems like it memoizes the subdirs that I tried to mount as volumes and shows just them, so running with /d, then /d/subdir, then /d will show subdir as content in last step).

It works somewhat better for /c than for /d though... For /c I can correctly access /c/Users/kamil and all contents in subdir, for /d there is nothing at all. Maybe Docker Toolbox VM is not actually mounting D: and for C: maybe it's a different problem.

@sawyer-peterson

This comment has been minimized.

sawyer-peterson commented May 9, 2018

you_dont_need_a_license_to_drive_a_whale
After fighting with Windows and Docker for a bit, I found out the secret to mapping the drives is as follows:
Update Docker so that it has access to the drive in question (see screenshot), and begin the path using /[drive] (in this case, /c/whatever)

@kskalski

This comment has been minimized.

kskalski commented May 11, 2018

Ah, ok. For Docker Toolbox (using VirtualBox) you can actually map drives (host C: or D:) under (guest, visible to docker containers) /mnt/c /mnt/d directories, so the workarounds mentioned in this issue could be skipped and the problem solved through VM settings.

@konczak

This comment has been minimized.

konczak commented Jun 12, 2018

Hello!
Yesterday I've had same issue with mounting folders with Docker-compose and this tip has helped me:
docker/compose#4303 (comment)

it was just as simple as "Reset credentials" in Docker "Shared drives", update password to current and restart Docker (in reset menu).

Maybe it will help somebody too.

@repos-jmp

This comment has been minimized.

repos-jmp commented Aug 2, 2018

I faced the same issue with docker-compose as well, by default Docker Toolbox for Windows only has access to C:\Users. Additional directories can be shared via VirtualBox UI.
REF : https://docs.docker.com/toolbox/toolbox_install_windows/#optional-add-shared-directories

@Ciantic

This comment has been minimized.

Ciantic commented Aug 14, 2018

I think the bind trick is a hack, this should be made to work:

     volumes:
     - "./db:/var/db"

Having to bind the drive first, then navigating to that new fake drive, makes things really complicated. If you already have a development setup with scripts that run the docker-compose, this is annoying thing. I can't go and edit the development scripts because they are correct.

Right now I should do a new script just for WSL, and that is clunky.

@mscraigloewen

This comment has been minimized.

Member

mscraigloewen commented Sep 6, 2018

Please refer to wsl.conf to help with your configuration settings. You may also find this post on using Docker with WSL to be useful

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment