Windows Containers - Symlink commands result in Access denied

[jasonbivins]

Description
Access denied when interacting with Symlinks inside windows containers

Steps to reproduce the issue:
1.start a windows container with a volume mounted
docker run -it -v C:\app:C:\app microsoft/nanoserver-insider

2. interact with or create a symlink
   cd to the C:\app folder
   mklink mylink file.txt
   Access is Denied

Output of docker version:

C:\app>docker version
Client:
 Version:       17.06.2-ee-10
 API version:   1.30
 Go version:    go1.8.7
 Git commit:    66261a0
 Built: Fri Apr 27 00:42:30 2018
 OS/Arch:       windows/amd64

Server:
 Engine:
  Version:      17.06.2-ee-10
  API version:  1.30 (minimum version 1.24)
  Go version:   go1.8.7
  Git commit:   66261a0
  Built:        Fri Apr 27 00:54:58 2018
  OS/Arch:      windows/amd64
  Experimental: false

Output of docker info:

C:\app>docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 17.06.2-ee-10
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.2214.amd64fre.rs1_release_1.180402-1758)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 16GiB
Name: Server2016
ID: KWWY:6A34:YZEL:WC2Q:EYCD:5UPU:HNTS:TOAO:F6B6:63OU:KPOR:6ZIB
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Originally reported here for the insider preview versions, but I can repro in Server 2016 LTSB and in 1709
docker/for-win#1997

[ielcoro]

Thanks @jasonbivins for the attention.

[el-gee]

I believe I have also run into this problem, but I feel a bit over my head and can't fully parse through all the layers of errors to be sure. A lot of stuff seems to just silently die off. It's like the stack traces don't always get fully propagated, perhaps people didn't code for the "no sym link support" edge case.

I'm also confused how such a seemingly large bug could have so little attention and others experiencing it. In my experience so far, if you're the only one having a problem the issue is you, not the code. So I really want to say that I just have some incorrect configuration, but the test above seems to pretty definitively demonstrate the issue.

Thus, at minimum, I also can confirm that on win 10 1803, using the latest 1803 and 1709 windowservercore images, I can reproduce the access denied error with the trivial example above. Interestingly, when I ran a linux container on the same windows host I was able to run something like ln -s File.txt Link from within the linux container. I could not only perform the command but I could also interact with the files as normal (cat Link would show the contents of File.txt). The file and link even appeared on the host through file explorer in the docker volume _data directory. However, the high level behavior I was originally trying to preform (run a java app that makes sym links to organize stuff) still didn't work and seemed to have the same errors / bad behavior as it's windows cousin.

Does this high level failure seem reasonable from this low level bug? If so, is there anything I can do to help address the issue? Finally, are the any recommended temporary work-arounds?

[rhpijnacker]

Same here ...

[filipesilva]

I can still reproduce with 1903 (#38220 (comment)).

[skeptic-monkey]

This issue is making the use of CI in Windows docker container quite painful.

We are using Gitlab Runner with docker-windows executor, which mounts a volume containing our source repository in the CI windows container.

As it is impossible to create symlink, Flutter and Bazel can't run properly in the mounted source directory and we are forced to copy the whole source tree out of it before building anything.

A second issue, is that we can't use the caching mechanism for the same reason (mounted volume), which makes all our CI jobs extremely slow.

[thaJeztah]

IIUC, this problem occurs when running containers with Hyper-V isolation (not with process isolation?); there's some mention of this in microsoft/Windows-Containers#117, and a related (possible duplicate) in #40125.

From reading those, it looks like this is not a bug/issue in Docker but a limitation in Windows (or the base image / host combination).

@kevpar @katiewasnothere any ideas if there's workarounds for this, or if it's possible to fix this in Windows?

[StriderAJR]

Agree with @skeptic-monkey. We also use bazel build inside docker image toolchain. Inability to create symlinks blocks bazel build inside host folder. And we have to copy docker local build folder to host machine which is painfully slow.

[jheaff1]

@StriderAJR , @skeptic-monkey , FYI - you can prevent Bazel from generating symlinks by passing the --experimental_convenience_symlinks=ignore argument to bazel. I can successfully build an application using bazel within a mounted host folder

[StriderAJR]

Yes, I use flag --experimental_convenience_symlinks=ignore I suppose that you ran container on Windows Server host machine? Because the problem exists only when running Windows Server image on Windows Desktop host machine. пн, 11 июл. 2022 г., 13:04 jheaff1 ***@***.***>:

…

@StriderAJR <https://github.com/StriderAJR> , @skeptic-monkey <https://github.com/skeptic-monkey> , FYI - you can prevent Bazel from generating symlinks by passing the --experimental_convenience_symlinks=ignore argument to bazel. I can successfully build an application using bazel within a mounted host folder — Reply to this email directly, view it on GitHub <#37024 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPZK4ZYYXONL5T4663MNKDVTPIRTANCNFSM4E7C5RAQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jheaff1]

Yes, I use flag --experimental_convenience_symlinks=ignore I suppose that you ran container on Windows Server host machine? Because the problem exists only when running Windows Server image on Windows Desktop host machine. пн, 11 июл. 2022 г., 13:04 jheaff1 @.***>:
…

Nope I ran a container based on windows/servercore:20H2 on a Windows 10 21H1 PC

[thaJeztah]

Nope I ran a container based on windows/servercore:20H2 on a Windows 10 21H1 PC

I think (per the earlier comments) it depends on wether or not Hyper-V isolation is used, which may be needed if the container's base-image is built for a different / non-matching Windows kernel.

[jheaff1]

Yep I was running under hyper-v isolation and Bazel initially failed due to the fact it couldn’t create a symlink.

[mg-aceik]

I can repro this issue still. Would be great to get a fix.

Host: Windows 11 Pro
Image: mcr.microsoft.com/windows/nanoserver:ltsc2022
Isolation: Process

Output of docker version:

Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.2
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Thu Feb  1 00:24:09 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.27.1 (136059)
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:01 2024
  OS/Arch:          windows/amd64
  Experimental:     false

I have tried
USER "NT Authority\System"
and
USER ContainerAdministrator