New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Symlinks don't work as expected #109

Closed
Morgy93 opened this Issue Sep 28, 2016 · 20 comments

Comments

Projects
None yet
@Morgy93

Morgy93 commented Sep 28, 2016

Expected behavior

I expect symlinks created from outside of the container to work.

Actual behavior

Symlinks created from outside the container don't work.

Information

Steps to reproduce the behavior

  1. Clone the repo
  2. Place some data within the www/ folder (e.g. touch example.txt)
  3. Create some folder within the www/folder (e.g. mkdir somefolder)
  4. Within the somefolder/ folder: ln -s ../example.txt example.txt
  5. Start the containers: docker-compose up
  6. Go right into the fpm or nginx container: docker exec -it foldername_fpm_1 bash
  7. Try to open / show / whatever the somefolder/example.txt: cat /var/www/html/somefolder/example.txt
  8. See that you (more or less hopefully) get the same error as I get: cat: somefolder/example.txt: No such file or directory
@rn

This comment has been minimized.

Show comment
Hide comment
@rn

rn Sep 28, 2016

Contributor

thanks for the report. where do you create the symlink? on the host?
thanks

Contributor

rn commented Sep 28, 2016

thanks for the report. where do you create the symlink? on the host?
thanks

@Morgy93

This comment has been minimized.

Show comment
Hide comment
@Morgy93

Morgy93 Oct 4, 2016

@rneugeba Yeah, I created the symlinks on the host machine (via the Linux subsystem for Windows [Bash on Ubuntu on Windows]) and they do work fine on that one, but not within the docker container.
While I create the symlink in the container, it works within the container but not on the host system.

Morgy93 commented Oct 4, 2016

@rneugeba Yeah, I created the symlinks on the host machine (via the Linux subsystem for Windows [Bash on Ubuntu on Windows]) and they do work fine on that one, but not within the docker container.
While I create the symlink in the container, it works within the container but not on the host system.

@rn

This comment has been minimized.

Show comment
Hide comment
@rn

rn Oct 4, 2016

Contributor

@Morgy93 thanks for getting back to me. Unfortunately, this doesn't work...

Native NTFS does not support symlinks. These are emulated with WSL by setting some file attribute (I think) and then add the target of the link in the file. CIFS/SMB on linux does something similar for emulating symlinks (see here), so does cygwin, as far as I know. MinGW handles symlinks different again.

Due to these different approaches we can only support symlinks created within containers. These then are resolvable in other containers but not on the host (and vice versa).

Unfortunately, I will have to close this as won't fix

Contributor

rn commented Oct 4, 2016

@Morgy93 thanks for getting back to me. Unfortunately, this doesn't work...

Native NTFS does not support symlinks. These are emulated with WSL by setting some file attribute (I think) and then add the target of the link in the file. CIFS/SMB on linux does something similar for emulating symlinks (see here), so does cygwin, as far as I know. MinGW handles symlinks different again.

Due to these different approaches we can only support symlinks created within containers. These then are resolvable in other containers but not on the host (and vice versa).

Unfortunately, I will have to close this as won't fix

@rn rn closed this Oct 4, 2016

@rn

This comment has been minimized.

Show comment
Hide comment
@rn

rn Oct 4, 2016

Contributor

@londoncalling could we add a section to the docs explaining the symlink behaviour?

Contributor

rn commented Oct 4, 2016

@londoncalling could we add a section to the docs explaining the symlink behaviour?

@Morgy93

This comment has been minimized.

Show comment
Hide comment
@Morgy93

Morgy93 Oct 4, 2016

@rneugeba Just to make it clear: There is no way to get working symlinks on the host and within the container at the same time?

Morgy93 commented Oct 4, 2016

@rneugeba Just to make it clear: There is no way to get working symlinks on the host and within the container at the same time?

@rn

This comment has been minimized.

Show comment
Hide comment
@rn

rn Oct 4, 2016

Contributor

@Morgy93 unfortunately, they use incompatible ways of emulating symlinks, so at the moment it is not possible.

Contributor

rn commented Oct 4, 2016

@Morgy93 unfortunately, they use incompatible ways of emulating symlinks, so at the moment it is not possible.

@londoncalling

This comment has been minimized.

Show comment
Hide comment
@londoncalling

londoncalling Oct 4, 2016

@rneugeba See docker/docker.github.io#34 (I tagged you on it, too).

londoncalling commented Oct 4, 2016

@rneugeba See docker/docker.github.io#34 (I tagged you on it, too).

@Morgy93

This comment has been minimized.

Show comment
Hide comment
@Morgy93

Morgy93 Oct 5, 2016

I've been seriously confused by this issue which got mostly resolved by reading this thread: https://forums.docker.com/t/symlinks-on-shared-volumes-not-supported/9288

I was sure that I already worked with symlinks on the Windows host side which also worked within the Linux vm / container - but, yeah.. sure... I used Docker Toolbox back then with Oracle VM VirtualBox and not the new Docker for Windows with Hyper-V.
All I need to figure out now is what this mfsymlinks is all about, but anyway - thanks for the information and help.

Morgy93 commented Oct 5, 2016

I've been seriously confused by this issue which got mostly resolved by reading this thread: https://forums.docker.com/t/symlinks-on-shared-volumes-not-supported/9288

I was sure that I already worked with symlinks on the Windows host side which also worked within the Linux vm / container - but, yeah.. sure... I used Docker Toolbox back then with Oracle VM VirtualBox and not the new Docker for Windows with Hyper-V.
All I need to figure out now is what this mfsymlinks is all about, but anyway - thanks for the information and help.

@rn

This comment has been minimized.

Show comment
Hide comment
@rn

rn Oct 5, 2016

Contributor

@Morgy93 glad that the explanation helped a little. One thing you can try, which may help to understand it better, is the following.

  • create a symlink in WSL and then look at it with powershell. the symlinks are just text files with some special characters and the location to the file being linked to.
  • create a symlink on a share volume in a container. again look at the contents of the file on the host, say with powershell.
Contributor

rn commented Oct 5, 2016

@Morgy93 glad that the explanation helped a little. One thing you can try, which may help to understand it better, is the following.

  • create a symlink in WSL and then look at it with powershell. the symlinks are just text files with some special characters and the location to the file being linked to.
  • create a symlink on a share volume in a container. again look at the contents of the file on the host, say with powershell.
@Morgy93

This comment has been minimized.

Show comment
Hide comment
@Morgy93

Morgy93 commented Dec 5, 2016

@rneugeba Will this resolve the issue?

https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/

@janhartigan

This comment has been minimized.

Show comment
Hide comment
@janhartigan

janhartigan Jan 30, 2017

@rneugeba any word on the above?

janhartigan commented Jan 30, 2017

@rneugeba any word on the above?

@smallscript

This comment has been minimized.

Show comment
Hide comment
@smallscript

smallscript Mar 30, 2017

NTFS not only supports symlinks, it does so with more control than on linux (ext family) and bsd (incl osx family). However, it is "poorly" understood because the web is littered with FUD about security issues.

NTFS supports forks (aka ADS - alternate data streams) which can and are used extensively in the OS and by many tools to also provide xattr capabilities. See Tuxera link here for "tag, xattr, stream/fork" discussion on referencing and using NTFS partitions on Linux and MacOS.

NTFS/Windows supports five forms of links (with symlinks implemented as ntfs-reparse-points).
a) Hard links for files (but not directories) that live on the same volume partition.
b) Exported (global) "soft" links termed as directory junctions (a type of "ntfs-reparse-point" only for directories not files)
c) Imported (local) links termed simply as "directory or file symbolic link" (a type of "ntfs-reparse-point")
d) LNK files termed simply as shell "shortcuts" (contain paths and can be tracked and managed by the OS)
e) URL files termed simply as "browser URL files" (portably work as URI link form using file://....).

NTFS is a journaled file-system (USN operations) so one can build or install services that monitor changes to the file system for various purposes including tracking any form of file-system changes.

Creating symlinks of the form (b) or (c) require admin rights (although downward relative path links shouldn't ever require that).

fsutil can be used with R2R, R2L, L2R, L2L to control resolution access of symbolic links referenced through SAMBA/SMB/CIFS mounted NTFS volumes.

When symlinks are viewed locally (same machine as the ntfs/volume is located on) then a JunctionLink and a DirSymLink to a Directory behave equivalently.

When symlinks are viewed remotely (as network mounted volume) a JunctionLink (/J) will be resolved on the remote-machine first, whereas a DirSymLink (/D) will always be resolved locally after the link is seen. Thus the export/import associated with their usage (Jn for export-usage, and Dir for import-usage).

NTFS/Windows uses drive letters so that can make the definition of absolute and relative target paths a little more complex to understand at first look. I.e., a target path of C:\... is obviously absolute. But, if you made a link on drive C:\foo and set the target to "\" the "\" is also absolute, just implicitly on C.

To be a relative path there must be no leading drive-letter and no leading "\" (aka "/"). Windows/NTFS namespace (UNC) paths actually are \\?... format and the NT subsystems use reparse points to create drive letters in the NT object-space. (see sysinternals tools for easy way to view the object-space WinObj tool)

A JunctionLink is only for "directories". In practice, JunctionLinks (a form of ntfs-reparse-point) can only reference and resolve local-volume-path-targets.

Thus JunctionLinks should really only be used with absolute target-paths and only when you want to export their target on a network drive. I.e., to export a link where its target is on different local volume-partition (same as hardlink rule) and you want that local-target to remotely and locally resolve to the same server-local-target volume-partition absolute path. I.e., you have a drive N:\foo - - > C:\bar-path; and you want local and remote users accessing the network-server's "N:\foo" directory to see the network server's "C:\bar-path" contents. If you don’t use a (/J) link, but use a (/D) link, the remote users will find nework-server's "N:\foo" confusingly resolving as their local "C:\bar-path".

The same rule, just less obvious, applies when using root-level references (absolute paths) with no drive letter appear on the same-local-volume. I.e., for when you want to export (invisibly resolving the link-target-path locally before sharing) the path to a remote network drive-share clients.

Like, say, you have a network share volume "N:\" and you want to export a path on some part of drive "C:\foo\path" (as a link target), then and only then does a (/J) JunctionLink make sense and is in fact the only way to do that (resolve an absolute path on a remote machine). If you used a (/D) dir-symlink with an absolute path on a network-drive, you would get quickly confused as a remote consumer of that network drive since it would try to find that absolute path on your local machine and not the remote machine.

You also need to use junctions on the same drive when the path cannot be made "relative" but is in fact absolute relative to the root of the same drive. So from "N:\foo\bar" to "N:\abc\def" you cannot make a valid absolute "\..." or relative "..\..." DirSymLink without issues, you need a JunctionLink of the form "\abc\def".

In all other cases, you want to use an NTFS DirSymLink or FileSymLink (same type of ntfs-reparse-point just applied to a directory vs a file).

See mklink command for basic usage from cli (command line). Also see dir command for displaying symlink targets, the "L" attribute, and also the /R for viewing xattr/forks on files and directories.

See also: wim services and 7z support for creating wim files, which are useful for archiving/zip/copying and restoring directory structures properly that contain symlinks. 7z a -snl -sns archive.wim source-files....

More detailed information on NTFS reparse-points and NT naming rules (including commentary on lxss file mechanism - linux-beta NT subsystem for Windows 10) can readily be found googling with the appropriate keywords used in this sentence and above paragraphs.

David Simmons (afm-scm.org / thelightphone.com)

smallscript commented Mar 30, 2017

NTFS not only supports symlinks, it does so with more control than on linux (ext family) and bsd (incl osx family). However, it is "poorly" understood because the web is littered with FUD about security issues.

NTFS supports forks (aka ADS - alternate data streams) which can and are used extensively in the OS and by many tools to also provide xattr capabilities. See Tuxera link here for "tag, xattr, stream/fork" discussion on referencing and using NTFS partitions on Linux and MacOS.

NTFS/Windows supports five forms of links (with symlinks implemented as ntfs-reparse-points).
a) Hard links for files (but not directories) that live on the same volume partition.
b) Exported (global) "soft" links termed as directory junctions (a type of "ntfs-reparse-point" only for directories not files)
c) Imported (local) links termed simply as "directory or file symbolic link" (a type of "ntfs-reparse-point")
d) LNK files termed simply as shell "shortcuts" (contain paths and can be tracked and managed by the OS)
e) URL files termed simply as "browser URL files" (portably work as URI link form using file://....).

NTFS is a journaled file-system (USN operations) so one can build or install services that monitor changes to the file system for various purposes including tracking any form of file-system changes.

Creating symlinks of the form (b) or (c) require admin rights (although downward relative path links shouldn't ever require that).

fsutil can be used with R2R, R2L, L2R, L2L to control resolution access of symbolic links referenced through SAMBA/SMB/CIFS mounted NTFS volumes.

When symlinks are viewed locally (same machine as the ntfs/volume is located on) then a JunctionLink and a DirSymLink to a Directory behave equivalently.

When symlinks are viewed remotely (as network mounted volume) a JunctionLink (/J) will be resolved on the remote-machine first, whereas a DirSymLink (/D) will always be resolved locally after the link is seen. Thus the export/import associated with their usage (Jn for export-usage, and Dir for import-usage).

NTFS/Windows uses drive letters so that can make the definition of absolute and relative target paths a little more complex to understand at first look. I.e., a target path of C:\... is obviously absolute. But, if you made a link on drive C:\foo and set the target to "\" the "\" is also absolute, just implicitly on C.

To be a relative path there must be no leading drive-letter and no leading "\" (aka "/"). Windows/NTFS namespace (UNC) paths actually are \\?... format and the NT subsystems use reparse points to create drive letters in the NT object-space. (see sysinternals tools for easy way to view the object-space WinObj tool)

A JunctionLink is only for "directories". In practice, JunctionLinks (a form of ntfs-reparse-point) can only reference and resolve local-volume-path-targets.

Thus JunctionLinks should really only be used with absolute target-paths and only when you want to export their target on a network drive. I.e., to export a link where its target is on different local volume-partition (same as hardlink rule) and you want that local-target to remotely and locally resolve to the same server-local-target volume-partition absolute path. I.e., you have a drive N:\foo - - > C:\bar-path; and you want local and remote users accessing the network-server's "N:\foo" directory to see the network server's "C:\bar-path" contents. If you don’t use a (/J) link, but use a (/D) link, the remote users will find nework-server's "N:\foo" confusingly resolving as their local "C:\bar-path".

The same rule, just less obvious, applies when using root-level references (absolute paths) with no drive letter appear on the same-local-volume. I.e., for when you want to export (invisibly resolving the link-target-path locally before sharing) the path to a remote network drive-share clients.

Like, say, you have a network share volume "N:\" and you want to export a path on some part of drive "C:\foo\path" (as a link target), then and only then does a (/J) JunctionLink make sense and is in fact the only way to do that (resolve an absolute path on a remote machine). If you used a (/D) dir-symlink with an absolute path on a network-drive, you would get quickly confused as a remote consumer of that network drive since it would try to find that absolute path on your local machine and not the remote machine.

You also need to use junctions on the same drive when the path cannot be made "relative" but is in fact absolute relative to the root of the same drive. So from "N:\foo\bar" to "N:\abc\def" you cannot make a valid absolute "\..." or relative "..\..." DirSymLink without issues, you need a JunctionLink of the form "\abc\def".

In all other cases, you want to use an NTFS DirSymLink or FileSymLink (same type of ntfs-reparse-point just applied to a directory vs a file).

See mklink command for basic usage from cli (command line). Also see dir command for displaying symlink targets, the "L" attribute, and also the /R for viewing xattr/forks on files and directories.

See also: wim services and 7z support for creating wim files, which are useful for archiving/zip/copying and restoring directory structures properly that contain symlinks. 7z a -snl -sns archive.wim source-files....

More detailed information on NTFS reparse-points and NT naming rules (including commentary on lxss file mechanism - linux-beta NT subsystem for Windows 10) can readily be found googling with the appropriate keywords used in this sentence and above paragraphs.

David Simmons (afm-scm.org / thelightphone.com)

@asteinlein

This comment has been minimized.

Show comment
Hide comment
@asteinlein

asteinlein May 7, 2017

Is this still "won't fix", even though symlinks is supported fine on Windows as explained by @smallscript?

The situation was even improved on Windows Creators Update, which allows creation of symlinks without admin privileges. Additionally, creating symlinks within WSL now also works as expected, creating a working symlink on the NTFS-side as well.

I'm cloning a git repo in WSL, which is working fine with working symlinks in Windows itself. However, using this in Docker fails as described by OP. Really a bummer if this won't be fixed.

asteinlein commented May 7, 2017

Is this still "won't fix", even though symlinks is supported fine on Windows as explained by @smallscript?

The situation was even improved on Windows Creators Update, which allows creation of symlinks without admin privileges. Additionally, creating symlinks within WSL now also works as expected, creating a working symlink on the NTFS-side as well.

I'm cloning a git repo in WSL, which is working fine with working symlinks in Windows itself. However, using this in Docker fails as described by OP. Really a bummer if this won't be fixed.

@rn

This comment has been minimized.

Show comment
Hide comment
@rn

rn May 7, 2017

Contributor

this also needs to be supported by the SMB client on Linux

Contributor

rn commented May 7, 2017

this also needs to be supported by the SMB client on Linux

@tobico

This comment has been minimized.

Show comment
Hide comment
@tobico

tobico May 20, 2017

Please consider reopening this issue. This will be a barrier to the adoption of Docker for Windows, as the same developers that would be inclined to use Docker for Windows will also be highly like to use WSL, and require these two tools to operate on the same files in a compatible manner.

tobico commented May 20, 2017

Please consider reopening this issue. This will be a barrier to the adoption of Docker for Windows, as the same developers that would be inclined to use Docker for Windows will also be highly like to use WSL, and require these two tools to operate on the same files in a compatible manner.

@Gonkers

This comment has been minimized.

Show comment
Hide comment
@Gonkers

Gonkers May 24, 2018

I think ran into this today. We are a mixed shop of windows, mac, linux. I use Windows on my dev box and have no issues using symlinks that are copied from our git repository. In fact docker-compose mounting the repository into the image works great too with symlinks. But when I attempt to docker build the image with source, the COPY command screws up the symlinks and breaks everything.

Gonkers commented May 24, 2018

I think ran into this today. We are a mixed shop of windows, mac, linux. I use Windows on my dev box and have no issues using symlinks that are copied from our git repository. In fact docker-compose mounting the repository into the image works great too with symlinks. But when I attempt to docker build the image with source, the COPY command screws up the symlinks and breaks everything.

@bartdeboer

This comment has been minimized.

Show comment
Hide comment
@bartdeboer

bartdeboer May 27, 2018

Run the Docker Quickstart Terminal as Administrator (and restart your docker-machine from there) for working symlinks. (Thank you JHipster: https://www.jhipster.tech/installation/)

bartdeboer commented May 27, 2018

Run the Docker Quickstart Terminal as Administrator (and restart your docker-machine from there) for working symlinks. (Thank you JHipster: https://www.jhipster.tech/installation/)

@ellyxc

This comment has been minimized.

Show comment
Hide comment
@ellyxc

ellyxc Jul 17, 2018

I having another issue with the symlinks. I am developing website using Yii where the assets files are using symlinks. The symlinks are working find inside the container (it doesn't matter if the symlinks aren't accessible from windows host). When opening the website in the browser i get 403 error message when downloading the assets files, but return 200 when trying to open the assets files in new tab. It seems there is performance issue here. Has anyone having the same issue?

ellyxc commented Jul 17, 2018

I having another issue with the symlinks. I am developing website using Yii where the assets files are using symlinks. The symlinks are working find inside the container (it doesn't matter if the symlinks aren't accessible from windows host). When opening the website in the browser i get 403 error message when downloading the assets files, but return 200 when trying to open the assets files in new tab. It seems there is performance issue here. Has anyone having the same issue?

@hniirane

This comment has been minimized.

Show comment
Hide comment
@hniirane

hniirane Aug 3, 2018

I having another issue with the symlinks. I am developing website using Yii where the assets files are using symlinks. The symlinks are working find inside the container (it doesn't matter if the symlinks aren't accessible from windows host). When opening the website in the browser i get 403 error message when downloading the assets files, but return 200 when trying to open the assets files in new tab. It seems there is performance issue here. Has anyone having the same issue?

I'm having issues that might be similar to yours @ellyxc . I'm trying to use symlinks to link a library using composer symlinks inside vendor/ so that I can develop the application and the library in parallel. This setup works on OS X without issues and works inside the container (i.e. running PHPUnit tests is fine). However when I try to load the App on a browser I get 500's with autoloader crapping out and not finding the symlinked classes. Running the same setup without the symlinks (i.e. library installed to vendors normally) there are no such issues.

The curious thing is that if I try to run the same requests via curl I it randomly works and randomly fails.

My current working theory is that somewhere in the share the symlinks resolve slowly when the access comes via nginx and that causes issues with the autoloader resolving classes. I will look into this in more detail next week as it's blocking our workflow on Windows.

hniirane commented Aug 3, 2018

I having another issue with the symlinks. I am developing website using Yii where the assets files are using symlinks. The symlinks are working find inside the container (it doesn't matter if the symlinks aren't accessible from windows host). When opening the website in the browser i get 403 error message when downloading the assets files, but return 200 when trying to open the assets files in new tab. It seems there is performance issue here. Has anyone having the same issue?

I'm having issues that might be similar to yours @ellyxc . I'm trying to use symlinks to link a library using composer symlinks inside vendor/ so that I can develop the application and the library in parallel. This setup works on OS X without issues and works inside the container (i.e. running PHPUnit tests is fine). However when I try to load the App on a browser I get 500's with autoloader crapping out and not finding the symlinked classes. Running the same setup without the symlinks (i.e. library installed to vendors normally) there are no such issues.

The curious thing is that if I try to run the same requests via curl I it randomly works and randomly fails.

My current working theory is that somewhere in the share the symlinks resolve slowly when the access comes via nginx and that causes issues with the autoloader resolving classes. I will look into this in more detail next week as it's blocking our workflow on Windows.

@hniirane

This comment has been minimized.

Show comment
Hide comment
@hniirane

hniirane Aug 10, 2018

@ellyxc I've investigated this for quite a while this week and I think you might be hitting the same issue as we are. Which is that files randomly appear not to exists when they are refered across symlinks on high I/O load (like loading a site in a browser that loads lot of stuff in parallel): #2396

Unfortunately I have no workaround that would apply to your scenario.

hniirane commented Aug 10, 2018

@ellyxc I've investigated this for quite a while this week and I think you might be hitting the same issue as we are. Which is that files randomly appear not to exists when they are refered across symlinks on high I/O load (like loading a site in a browser that loads lot of stuff in parallel): #2396

Unfortunately I have no workaround that would apply to your scenario.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment