Symlinks don't work as expected #109

Closed
opened this Issue Sep 28, 2016 · 20 comments

Projects
None yet

Expected behavior

I expect symlinks created from outside of the container to work.

Actual behavior

Symlinks created from outside the container don't work.

Steps to reproduce the behavior

1. Clone the repo
2. Place some data within the www/ folder (e.g. touch example.txt)
3. Create some folder within the www/folder (e.g. mkdir somefolder)
4. Within the somefolder/ folder: ln -s ../example.txt example.txt
5. Start the containers: docker-compose up
6. Go right into the fpm or nginx container: docker exec -it foldername_fpm_1 bash
7. Try to open / show / whatever the somefolder/example.txt: cat /var/www/html/somefolder/example.txt
8. See that you (more or less hopefully) get the same error as I get: cat: somefolder/example.txt: No such file or directory
Contributor

rn commented Sep 28, 2016

 thanks for the report. where do you create the symlink? on the host? thanks

Morgy93 commented Oct 4, 2016

 @rneugeba Yeah, I created the symlinks on the host machine (via the Linux subsystem for Windows [Bash on Ubuntu on Windows]) and they do work fine on that one, but not within the docker container. While I create the symlink in the container, it works within the container but not on the host system.
Contributor

rn commented Oct 4, 2016

 @Morgy93 thanks for getting back to me. Unfortunately, this doesn't work... Native NTFS does not support symlinks. These are emulated with WSL by setting some file attribute (I think) and then add the target of the link in the file. CIFS/SMB on linux does something similar for emulating symlinks (see here), so does cygwin, as far as I know. MinGW handles symlinks different again. Due to these different approaches we can only support symlinks created within containers. These then are resolvable in other containers but not on the host (and vice versa). Unfortunately, I will have to close this as won't fix

Contributor

rn commented Oct 4, 2016

 @londoncalling could we add a section to the docs explaining the symlink behaviour?

Morgy93 commented Oct 4, 2016

 @rneugeba Just to make it clear: There is no way to get working symlinks on the host and within the container at the same time?
Contributor

rn commented Oct 4, 2016

 @Morgy93 unfortunately, they use incompatible ways of emulating symlinks, so at the moment it is not possible.

Merged

londoncalling commented Oct 4, 2016

 @rneugeba See docker/docker.github.io#34 (I tagged you on it, too).

Morgy93 commented Oct 5, 2016

 I've been seriously confused by this issue which got mostly resolved by reading this thread: https://forums.docker.com/t/symlinks-on-shared-volumes-not-supported/9288 I was sure that I already worked with symlinks on the Windows host side which also worked within the Linux vm / container - but, yeah.. sure... I used Docker Toolbox back then with Oracle VM VirtualBox and not the new Docker for Windows with Hyper-V. All I need to figure out now is what this mfsymlinks is all about, but anyway - thanks for the information and help.
Contributor

rn commented Oct 5, 2016

 @Morgy93 glad that the explanation helped a little. One thing you can try, which may help to understand it better, is the following. create a symlink in WSL and then look at it with powershell. the symlinks are just text files with some special characters and the location to the file being linked to. create a symlink on a share volume in a container. again look at the contents of the file on the host, say with powershell.

Merged

Morgy93 commented Dec 5, 2016

 @rneugeba Will this resolve the issue? https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/

janhartigan commented Jan 30, 2017

 @rneugeba any word on the above?

smallscript commented Mar 30, 2017 • edited

 NTFS not only supports symlinks, it does so with more control than on linux (ext family) and bsd (incl osx family). However, it is "poorly" understood because the web is littered with FUD about security issues. NTFS supports forks (aka ADS - alternate data streams) which can and are used extensively in the OS and by many tools to also provide xattr capabilities. See Tuxera link here for "tag, xattr, stream/fork" discussion on referencing and using NTFS partitions on Linux and MacOS. NTFS/Windows supports five forms of links (with symlinks implemented as ntfs-reparse-points). a) Hard links for files (but not directories) that live on the same volume partition. b) Exported (global) "soft" links termed as directory junctions (a type of "ntfs-reparse-point" only for directories not files) c) Imported (local) links termed simply as "directory or file symbolic link" (a type of "ntfs-reparse-point") d) LNK files termed simply as shell "shortcuts" (contain paths and can be tracked and managed by the OS) e) URL files termed simply as "browser URL files" (portably work as URI link form using file://....). NTFS is a journaled file-system (USN operations) so one can build or install services that monitor changes to the file system for various purposes including tracking any form of file-system changes. Creating symlinks of the form (b) or (c) require admin rights (although downward relative path links shouldn't ever require that). fsutil can be used with R2R, R2L, L2R, L2L to control resolution access of symbolic links referenced through SAMBA/SMB/CIFS mounted NTFS volumes. When symlinks are viewed locally (same machine as the ntfs/volume is located on) then a JunctionLink and a DirSymLink to a Directory behave equivalently. When symlinks are viewed remotely (as network mounted volume) a JunctionLink (/J) will be resolved on the remote-machine first, whereas a DirSymLink (/D) will always be resolved locally after the link is seen. Thus the export/import associated with their usage (Jn for export-usage, and Dir for import-usage). NTFS/Windows uses drive letters so that can make the definition of absolute and relative target paths a little more complex to understand at first look. I.e., a target path of C:\... is obviously absolute. But, if you made a link on drive C:\foo and set the target to "\" the "\" is also absolute, just implicitly on C. To be a relative path there must be no leading drive-letter and no leading "\" (aka "/"). Windows/NTFS namespace (UNC) paths actually are \\?... format and the NT subsystems use reparse points to create drive letters in the NT object-space. (see sysinternals tools for easy way to view the object-space WinObj tool) A JunctionLink is only for "directories". In practice, JunctionLinks (a form of ntfs-reparse-point) can only reference and resolve local-volume-path-targets. Thus JunctionLinks should really only be used with absolute target-paths and only when you want to export their target on a network drive. I.e., to export a link where its target is on different local volume-partition (same as hardlink rule) and you want that local-target to remotely and locally resolve to the same server-local-target volume-partition absolute path. I.e., you have a drive N:\foo - - > C:\bar-path; and you want local and remote users accessing the network-server's "N:\foo" directory to see the network server's "C:\bar-path" contents. If you don’t use a (/J) link, but use a (/D) link, the remote users will find nework-server's "N:\foo" confusingly resolving as their local "C:\bar-path". The same rule, just less obvious, applies when using root-level references (absolute paths) with no drive letter appear on the same-local-volume. I.e., for when you want to export (invisibly resolving the link-target-path locally before sharing) the path to a remote network drive-share clients. Like, say, you have a network share volume "N:\" and you want to export a path on some part of drive "C:\foo\path" (as a link target), then and only then does a (/J) JunctionLink make sense and is in fact the only way to do that (resolve an absolute path on a remote machine). If you used a (/D) dir-symlink with an absolute path on a network-drive, you would get quickly confused as a remote consumer of that network drive since it would try to find that absolute path on your local machine and not the remote machine. You also need to use junctions on the same drive when the path cannot be made "relative" but is in fact absolute relative to the root of the same drive. So from "N:\foo\bar" to "N:\abc\def" you cannot make a valid absolute "\..." or relative "..\..." DirSymLink without issues, you need a JunctionLink of the form "\abc\def". In all other cases, you want to use an NTFS DirSymLink or FileSymLink (same type of ntfs-reparse-point just applied to a directory vs a file). See mklink command for basic usage from cli (command line). Also see dir command for displaying symlink targets, the "L" attribute, and also the /R for viewing xattr/forks on files and directories. See also: wim services and 7z support for creating wim files, which are useful for archiving/zip/copying and restoring directory structures properly that contain symlinks. 7z a -snl -sns archive.wim source-files.... More detailed information on NTFS reparse-points and NT naming rules (including commentary on lxss file mechanism - linux-beta NT subsystem for Windows 10) can readily be found googling with the appropriate keywords used in this sentence and above paragraphs. David Simmons (afm-scm.org / thelightphone.com)

asteinlein commented May 7, 2017

 Is this still "won't fix", even though symlinks is supported fine on Windows as explained by @smallscript? The situation was even improved on Windows Creators Update, which allows creation of symlinks without admin privileges. Additionally, creating symlinks within WSL now also works as expected, creating a working symlink on the NTFS-side as well. I'm cloning a git repo in WSL, which is working fine with working symlinks in Windows itself. However, using this in Docker fails as described by OP. Really a bummer if this won't be fixed.
Contributor

rn commented May 7, 2017

 this also needs to be supported by the SMB client on Linux

tobico commented May 20, 2017

 Please consider reopening this issue. This will be a barrier to the adoption of Docker for Windows, as the same developers that would be inclined to use Docker for Windows will also be highly like to use WSL, and require these two tools to operate on the same files in a compatible manner.

Gonkers commented May 24, 2018

 I think ran into this today. We are a mixed shop of windows, mac, linux. I use Windows on my dev box and have no issues using symlinks that are copied from our git repository. In fact docker-compose mounting the repository into the image works great too with symlinks. But when I attempt to docker build the image with source, the COPY command screws up the symlinks and breaks everything.

bartdeboer commented May 27, 2018

 Run the Docker Quickstart Terminal as Administrator (and restart your docker-machine from there) for working symlinks. (Thank you JHipster: https://www.jhipster.tech/installation/)

ellyxc commented Jul 17, 2018

 I having another issue with the symlinks. I am developing website using Yii where the assets files are using symlinks. The symlinks are working find inside the container (it doesn't matter if the symlinks aren't accessible from windows host). When opening the website in the browser i get 403 error message when downloading the assets files, but return 200 when trying to open the assets files in new tab. It seems there is performance issue here. Has anyone having the same issue?