Symlinks don't work as expected

[Morgy93]

Expected behavior

I expect symlinks created from outside of the container to work.

Actual behavior

Symlinks created from outside the container don't work.

Information

• Diagnostic ID: EF5CEE54-6553-4655-A810-98A1549E5A7D/2016-09-28_13-38-06
• Repo: https://github.com/werfu/magento-docker-compose-dev
• Docker Version 1.12.1-stable (build: 7135) 995236d
• Windows 10 Pro Version 1607 Build 14393.187

Steps to reproduce the behavior

1. Clone the repo
2. Place some data within the www/ folder (e.g. touch example.txt)
3. Create some folder within the www/folder (e.g. mkdir somefolder)
4. Within the somefolder/ folder: ln -s ../example.txt example.txt
5. Start the containers: docker-compose up
6. Go right into the fpm or nginx container: docker exec -it foldername_fpm_1 bash
7. Try to open / show / whatever the somefolder/example.txt: cat /var/www/html/somefolder/example.txt
8. See that you (more or less hopefully) get the same error as I get: cat: somefolder/example.txt: No such file or directory

[rn]

thanks for the report. where do you create the symlink? on the host?
thanks

[Morgy93]

@rneugeba Yeah, I created the symlinks on the host machine (via the Linux subsystem for Windows [Bash on Ubuntu on Windows]) and they do work fine on that one, but not within the docker container.
While I create the symlink in the container, it works within the container but not on the host system.

[rn]

@Morgy93 thanks for getting back to me. Unfortunately, this doesn't work...

Native NTFS does not support symlinks. These are emulated with WSL by setting some file attribute (I think) and then add the target of the link in the file. CIFS/SMB on linux does something similar for emulating symlinks (see here), so does cygwin, as far as I know. MinGW handles symlinks different again.

Due to these different approaches we can only support symlinks created within containers. These then are resolvable in other containers but not on the host (and vice versa).

Unfortunately, I will have to close this as won't fix

[rn]

@londoncalling could we add a section to the docs explaining the symlink behaviour?

[Morgy93]

@rneugeba Just to make it clear: There is no way to get working symlinks on the host and within the container at the same time?

[rn]

@Morgy93 unfortunately, they use incompatible ways of emulating symlinks, so at the moment it is not possible.

[londoncalling]

@rneugeba See docker/docs#34 (I tagged you on it, too).

[Morgy93]

I've been seriously confused by this issue which got mostly resolved by reading this thread: https://forums.docker.com/t/symlinks-on-shared-volumes-not-supported/9288

I was sure that I already worked with symlinks on the Windows host side which also worked within the Linux vm / container - but, yeah.. sure... I used Docker Toolbox back then with Oracle VM VirtualBox and not the new Docker for Windows with Hyper-V.
All I need to figure out now is what this mfsymlinks is all about, but anyway - thanks for the information and help.

[rn]

@Morgy93 glad that the explanation helped a little. One thing you can try, which may help to understand it better, is the following.

• create a symlink in WSL and then look at it with powershell. the symlinks are just text files with some special characters and the location to the file being linked to.
• create a symlink on a share volume in a container. again look at the contents of the file on the host, say with powershell.

[Morgy93]

@rneugeba Will this resolve the issue?

https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/

[janhartigan]

@rneugeba any word on the above?

[smallscript]

NTFS not only supports symlinks, it does so with more control than on linux (ext family) and bsd (incl osx family). However, it is "poorly" understood because the web is littered with FUD about security issues.

NTFS supports forks (aka ADS - alternate data streams) which can and are used extensively in the OS and by many tools to also provide xattr capabilities. See Tuxera link here for "tag, xattr, stream/fork" discussion on referencing and using NTFS partitions on Linux and MacOS.

NTFS/Windows supports five forms of links (with symlinks implemented as ntfs-reparse-points).
a) Hard links for files (but not directories) that live on the same volume partition.
b) Exported (global) "soft" links termed as directory junctions (a type of "ntfs-reparse-point" only for directories not files)
c) Imported (local) links termed simply as "directory or file symbolic link" (a type of "ntfs-reparse-point")
d) LNK files termed simply as shell "shortcuts" (contain paths and can be tracked and managed by the OS)
e) URL files termed simply as "browser URL files" (portably work as URI link form using file://....).

NTFS is a journaled file-system (USN operations) so one can build or install services that monitor changes to the file system for various purposes including tracking any form of file-system changes.

Creating symlinks of the form (b) or (c) require admin rights (although downward relative path links shouldn't ever require that).

fsutil can be used with R2R, R2L, L2R, L2L to control resolution access of symbolic links referenced through SAMBA/SMB/CIFS mounted NTFS volumes.

When symlinks are viewed locally (same machine as the ntfs/volume is located on) then a JunctionLink and a DirSymLink to a Directory behave equivalently.

When symlinks are viewed remotely (as network mounted volume) a JunctionLink (/J) will be resolved on the remote-machine first, whereas a DirSymLink (/D) will always be resolved locally after the link is seen. Thus the export/import associated with their usage (Jn for export-usage, and Dir for import-usage).

NTFS/Windows uses drive letters so that can make the definition of absolute and relative target paths a little more complex to understand at first look. I.e., a target path of C:\... is obviously absolute. But, if you made a link on drive C:\foo and set the target to "\" the "\" is also absolute, just implicitly on C.

To be a relative path there must be no leading drive-letter and no leading "\" (aka "/"). Windows/NTFS namespace (UNC) paths actually are \\?... format and the NT subsystems use reparse points to create drive letters in the NT object-space. (see sysinternals tools for easy way to view the object-space WinObj tool)

A JunctionLink is only for "directories". In practice, JunctionLinks (a form of ntfs-reparse-point) can only reference and resolve local-volume-path-targets.

Thus JunctionLinks should really only be used with absolute target-paths and only when you want to export their target on a network drive. I.e., to export a link where its target is on different local volume-partition (same as hardlink rule) and you want that local-target to remotely and locally resolve to the same server-local-target volume-partition absolute path. I.e., you have a drive N:\foo - - > C:\bar-path; and you want local and remote users accessing the network-server's "N:\foo" directory to see the network server's "C:\bar-path" contents. If you don’t use a (/J) link, but use a (/D) link, the remote users will find nework-server's "N:\foo" confusingly resolving as their local "C:\bar-path".

The same rule, just less obvious, applies when using root-level references (absolute paths) with no drive letter appear on the same-local-volume. I.e., for when you want to export (invisibly resolving the link-target-path locally before sharing) the path to a remote network drive-share clients.

Like, say, you have a network share volume "N:\" and you want to export a path on some part of drive "C:\foo\path" (as a link target), then and only then does a (/J) JunctionLink make sense and is in fact the only way to do that (resolve an absolute path on a remote machine). If you used a (/D) dir-symlink with an absolute path on a network-drive, you would get quickly confused as a remote consumer of that network drive since it would try to find that absolute path on your local machine and not the remote machine.

You also need to use junctions on the same drive when the path cannot be made "relative" but is in fact absolute relative to the root of the same drive. So from "N:\foo\bar" to "N:\abc\def" you cannot make a valid absolute "\..." or relative "..\..." DirSymLink without issues, you need a JunctionLink of the form "\abc\def".

In all other cases, you want to use an NTFS DirSymLink or FileSymLink (same type of ntfs-reparse-point just applied to a directory vs a file).

See mklink command for basic usage from cli (command line). Also see dir command for displaying symlink targets, the "L" attribute, and also the /R for viewing xattr/forks on files and directories.

See also: wim services and 7z support for creating wim files, which are useful for archiving/zip/copying and restoring directory structures properly that contain symlinks. 7z a -snl -sns archive.wim source-files....

More detailed information on NTFS reparse-points and NT naming rules (including commentary on lxss file mechanism - linux-beta NT subsystem for Windows 10) can readily be found googling with the appropriate keywords used in this sentence and above paragraphs.

David Simmons (afm-scm.org / thelightphone.com)

[asteinlein]

Is this still "won't fix", even though symlinks is supported fine on Windows as explained by @smallscript?

The situation was even improved on Windows Creators Update, which allows creation of symlinks without admin privileges. Additionally, creating symlinks within WSL now also works as expected, creating a working symlink on the NTFS-side as well.

I'm cloning a git repo in WSL, which is working fine with working symlinks in Windows itself. However, using this in Docker fails as described by OP. Really a bummer if this won't be fixed.

[rn]

this also needs to be supported by the SMB client on Linux

[tobico]

Please consider reopening this issue. This will be a barrier to the adoption of Docker for Windows, as the same developers that would be inclined to use Docker for Windows will also be highly like to use WSL, and require these two tools to operate on the same files in a compatible manner.

[Gonkers]

I think ran into this today. We are a mixed shop of windows, mac, linux. I use Windows on my dev box and have no issues using symlinks that are copied from our git repository. In fact docker-compose mounting the repository into the image works great too with symlinks. But when I attempt to docker build the image with source, the COPY command screws up the symlinks and breaks everything.

[bartdeboer]

Run the Docker Quickstart Terminal as Administrator (and restart your docker-machine from there) for working symlinks. (Thank you JHipster: https://www.jhipster.tech/installation/)

[ellyxc]

I having another issue with the symlinks. I am developing website using Yii where the assets files are using symlinks. The symlinks are working find inside the container (it doesn't matter if the symlinks aren't accessible from windows host). When opening the website in the browser i get 403 error message when downloading the assets files, but return 200 when trying to open the assets files in new tab. It seems there is performance issue here. Has anyone having the same issue?

[hniirane]

I having another issue with the symlinks. I am developing website using Yii where the assets files are using symlinks. The symlinks are working find inside the container (it doesn't matter if the symlinks aren't accessible from windows host). When opening the website in the browser i get 403 error message when downloading the assets files, but return 200 when trying to open the assets files in new tab. It seems there is performance issue here. Has anyone having the same issue?

I'm having issues that might be similar to yours @ellyxc . I'm trying to use symlinks to link a library using composer symlinks inside vendor/ so that I can develop the application and the library in parallel. This setup works on OS X without issues and works inside the container (i.e. running PHPUnit tests is fine). However when I try to load the App on a browser I get 500's with autoloader crapping out and not finding the symlinked classes. Running the same setup without the symlinks (i.e. library installed to vendors normally) there are no such issues.

The curious thing is that if I try to run the same requests via curl I it randomly works and randomly fails.

My current working theory is that somewhere in the share the symlinks resolve slowly when the access comes via nginx and that causes issues with the autoloader resolving classes. I will look into this in more detail next week as it's blocking our workflow on Windows.

[hniirane]

@ellyxc I've investigated this for quite a while this week and I think you might be hitting the same issue as we are. Which is that files randomly appear not to exists when they are refered across symlinks on high I/O load (like loading a site in a browser that loads lot of stuff in parallel): #2396

Unfortunately I have no workaround that would apply to your scenario.

[safizn]

🤯 WSL symlinks are different from Linux VM or Linux container symlinks !
Why ? WSL is useless in such cases. Installing node_modules from within WSL creates symlinks in node_modules/.bin folder, that cannot be used within MobyLinux VM or Containers (Docker for Windows).

On the verge of switching my whole OS to Linux, I don't think sticking with Windows does for me any good anymore.

[gety9]

@myuseringithub what way did you go in the end ? have same situation but i am not eager to switch from Windows .

[safizn]

@gety9 I've redesigned my development/deployment workflow, I no longer use Containers for development, rather develop in a WSL Node JS installation. This simplified my workflow and made integrating with other tools easier. For production I use containers where the production environment is Linux so any symlinks would work without issues.

[wclr]

Docker is cool for dev workfows. Just don't use symlinks, use https://github.com/whitecolor/yalc for "linking" dev packages for example, and inside containers I use their own shared node_modules volumes where symlinks done by yarn workspaces work as expected.

[JeroenvdV]

If you land here from Google: there might be a solution.

The problem had is my container can't find files that are in git submodules if I use git from WSL. Actually, it was composer that made the symlink to the vendor folder.

The fix was to turn on Developer Mode in Windows Settings, then recreate the symlink (I deleted the relevant folder from the vendor folder and re-ran composer install).

[Slayvin]

@ellyxc @hniirane
I posted a workaround on SO : https://stackoverflow.com/questions/58857724/how-can-i-make-symlinks-made-from-inside-docker-linux-containers-to-be-seen-from/62330577#62330577

Basically, the trick is to use a volume to mount your vendor package parent directory , and keep the symlink made in Windows to the package directory itself.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked