hcsshim::ImportLayer failed after cygwin and vs2015 build tools installations

[isanych]

Description

I cannot install cygwin during docker build anymore (but I still can install and use it during docker run).

Steps to reproduce the issue:
Dockerfile:

FROM mcr.microsoft.com/windows/servercore:2004

RUN powershell -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
RUN choco feature enable -n=allowGlobalConfirmation
RUN choco feature disable -n=showDownloadProgress
RUN cinst cygwin

docker build .

Describe the results you received:

...
Step 5/5 : RUN cinst cygwin
 ---> Running in b00fa2c610e1
Chocolatey v0.10.15
Installing the following packages:
cygwin
By installing you accept licenses for the packages.

chocolatey-core.extension v1.3.5.1 [Approved]
chocolatey-core.extension package files install completed. Performing other installation steps.
 Installed/updated chocolatey-core extensions.
 The install of chocolatey-core.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\chocolatey-core'

Cygwin v3.1.5 [Approved]
cygwin package files install completed. Performing other installation steps.
Download site: http://mirrors.kernel.org/sourceware/cygwin/
Installing 64-bit Cygwin...
Cygwin has been installed.
Added C:\ProgramData\chocolatey\bin\Cygwin.exe shim pointed to 'c:\tools\cygwin\cygwin.bat'.
Copying cygwin package manager (setup) to C:\tools\cygwin
Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).
 ShimGen has successfully created a shim for setup-x86.exe
 The install of cygwin was successful.
  Software installed to 'C:\tools\cygwin'

Chocolatey installed 2/2 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Enjoy using Chocolatey? Explore more amazing features to take your
experience to the next level at
 https://chocolatey.org/compare
re-exec error: exit status 1: output: time="2020-06-02T18:00:44+01:00" level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" error="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" importFolderPath="D:\\Docker\\tmp\\hcs894249803" path="\\\\?\\D:\\Docker\\windowsfilter\\a0eea805d4dac2f148f2cb7f8a8620c96f22242d76a2ebb94d457745bdce7190"
hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)

Describe the results you expected:
Image created successfully, as it was happening for a long time.

Additional information you deem important (e.g. issue happens only occasionally):
Something changed in cygwin, as failure observed on hosts where it was installing successfully before and there were no changes in docker on these hosts. I see the same issue on docker 19.03.05, 19.03.08, 19.03.11 and master, on windows server 1903, windows server 2004 and windows 10 2004, using mcr.microsoft.com/windows:1903, mcr.microsoft.com/windows:2004 and mcr.microsoft.com/windows/servercore:2004 base images - I could not install cygwin anywhere anymore.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:23:10 2020
 OS/Arch:           windows/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.24)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:37:20 2020
  OS/Arch:          windows/amd64
  Experimental:     true

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 7
  Running: 0
  Paused: 0
  Stopped: 7
 Images: 33
 Server Version: 19.03.8
 Storage Driver: windowsfilter (windows) lcow (linux)
  Windows:
  LCOW:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 19041 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows 10 Pro Version 2004 (OS Build 19041.264)
 OSType: windows
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.92GiB
 Name: beta
 ID: KDC4:AXBE:IEZA:QYBJ:RDPR:YGDB:U6NR:7DSX:VQRN:L3IK:L7YI:W5J7
 Docker Root Dir: D:\Docker
 Debug Mode: true
  File Descriptors: -1
  Goroutines: 28
  System Time: 2020-06-02T18:03:58.8208563+01:00
  EventsListeners: 1
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical machines and Proxmox VMs, Windows 10 and Windows Server (SAC)

[isanych]

The same issue with vs2015 build tools which also was fine before:

FROM mcr.microsoft.com/windows/servercore:2004

RUN powershell -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
RUN choco feature enable -n=allowGlobalConfirmation
RUN choco feature disable -n=showDownloadProgress
RUN cinst visualcpp-build-tools --version 14.0.25420.1

error:

Step 5/5 : RUN cinst visualcpp-build-tools --version 14.0.25420.1
 ---> Running in e568f2064bc9
Chocolatey v0.10.15
Installing the following packages:
visualcpp-build-tools
By installing you accept licenses for the packages.

DotNet4.5.2 v4.5.2.20140902 [Approved]
dotnet4.5.2 package files install completed. Performing other installation steps.
Microsoft .Net 4.5.2 Framework is already installed on your machine.
 The install of dotnet4.5.2 was successful.
  Software install location not explicitly set, could be in package or
  default install location if installer.

visualcpp-build-tools v14.0.25420.1 [Approved]
visualcpp-build-tools package files install completed. Performing other installation steps.
Downloading visualcpp-build-tools
  from 'https://download.microsoft.com/download/5/f/7/5f7acaeb-8363-451f-9425-68a90f98b238/visualcppbuildtools_full.exe'

Download of visualcppbuildtools_full.exe (3.14 MB) completed.
Hashes match.
Installing visualcpp-build-tools...
visualcpp-build-tools has been installed.
  visualcpp-build-tools may be able to be automatically uninstalled.
Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).
 The install of visualcpp-build-tools was successful.
  Software installed as 'EXE', install location is likely default.

Chocolatey installed 2/2 packages.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
re-exec error: exit status 1: output: time="2020-06-04T10:20:54+01:00" level=error msg="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" error="hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)" importFolderPath="D:\\Docker\\tmp\\hcs957998503" path="\\\\?\\D:\\Docker\\windowsfilter\\1e239a85b16a7df406d83c9b8d1d5cb1be762e7cf6379d3c76334c3ce20e0e4f"
hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3)

[thaJeztah]

Something changed in cygwin, as failure observed on hosts where it was installing successfully before and there were no changes in docker on these hosts

Is the problem fixed if you install an older version? (I don't have much experience with chocolatey, but looks like choco install cygwin --version <some version> ....; https://chocolatey.org/docs/commandsinstall)

[isanych]

@thaJeztah I've tried to specify older version, but it has not changed anything (as cygwin installer downloading packages from the same repository anyway). For cygwin I'm currently using next ugly workaround - I've archived installed cygwin from images created earlier and extracting that archive - but what would I do when I'll need add new packages - I have no idea.

VS2015 even worse case - version already was specified in Dockerfile (but again it is only installer which download other stuff too) - and It is not easy to transfer it from older image, as there are multiple locations and registry involved.

[thaJeztah]

Is it possible your machines have had a Windows (security) update installed? I recall there were some problematic updates a while back; #40525 (comment)

[isanych]

I've tried 4 different machines, existing and freshly installed, physical and VMs, Windows 10 2004, Windows Server 2004, Windows Server 1903 - but windows update enabled on all machines - and multiple docker versions including master - I see the same failure everywhere.

[thaJeztah]

(I'm not a Windows user, so mainly trying to narrow-down things).
The :2004 (or :1903 tags for that matter) are "floating" tags, which means they will be updated periodically. If all machines have pulled the current version of the base image, It may be worth trying if picking an older version of the base image makes the issue go away as well https://hub.docker.com/_/microsoft-windows-servercore (linking to the docker hub page, but I think there's other places to get a list of older tags).

[thaJeztah]

@kevpar any ideas? More details that could be obtained to root-cause the problem?

[isanych]

I've just created new AWS instance based on Windows_Server-2004-English-Core-ContainersLatest-2020.05.27 - ami-0afc0c287d37358a7 (docker 19.03.05 (latest available on servers - docker/for-win#7014) preinstalled) and got the same error there

[isanych]

After checking kevpar's profile, repeating test on Azure :)
VM based on [smalldisk] Windows Server, version 2004 with Containers (docker 19.03.5 preinstalled) - issue reproduced there too.

[chandanpjain]

I am facing the same issue and we have tried on mcr.microsoft.com/windows/servercore:ltsc2019. Found similar issue in stackoverflow.

[exceed-alae]

I have same problem. (From: mcr.microsoft.com/dotnet/framework/sdk:4.8-windowsservercore-1909)
I had no problem using the same Dockerfile 2months ago.

[kempniu]

I have the same problem with Windows Server 2016 as the host system, Docker 19.03.5 (which is the latest version currently available for that system through DockerMsftProvider), and microsoft/dotnet-framework:3.5-sdk-windowsservercore-ltsc2016 as the base image - installing cygwin using choco install triggers a hcsshim::ImportLayer error during docker build. When run in a "live" container (created using docker run), the same command works fine, but subsequently running docker commit for such a container triggers the exact same error. The Dockerfile I am using was working fine back in January 2020 with the same Docker version.

I believe @thaJeztah's suggestion about the February 2020 Windows update is pertinent to this issue: I still keep an "old" Windows Server 2016 box around which contains a properly generated Docker image with Cygwin; after transferring that image to a "new" server, some binaries (e.g. msbuild.exe) just exit silently, which (I think) matches the symptoms described in Microsoft's article:

https://support.microsoft.com/en-us/help/4542617/you-might-encounter-issues-when-using-windows-server-containers-with-t

Things are still working just fine on the "old" server (it was provisioned in January 2020 and has not had any Windows updates installed since).

I tried using docker-ci-zap and starting the Docker daemon with --storage-opt size=50G to no avail.

I would be happy to do any testing that could help resolve this issue as it is currently preventing me from provisioning any new CI server.

[isanych]

@kempniu, unfortunately it is not the case. I've installed clean servers 1903 and 1909 (en_windows_server_version_1903_x64_dvd_58ddff4b.iso & en_windows_server_version_1909_x64_dvd_894c6446.iso) without any updates - docker build still failing.

So I'm still looking for workarounds for cygwin and vs2015 c++ build tools installation

[isanych]

Cygwin workaround which is working for me - locations and names emulating cinst cygwin:

md c:\tools\cygwin && cd c:\tools\cygwin && curl -Sso cygwinsetup.exe https://cygwin.com/setup-x86_64.exe && start /wait cygwinsetup -q --root C:\tools\cygwin -P dos2unix,make,perl,python27,python38,rsync,libxml2,pbzip2,mc -X --site http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/2020/05/31/142136/

So version from May 31st is ok, version from June 1st breaking docker.

[markmssd]

Thanks @isanych, that worked for me!

[haubi]

I'd wonder if it does work with cygwin version 3.1.4, as with cygwin-3.1.5 this bug was introduced, which turned out to be nontrivial: https://cygwin.com/pipermail/cygwin/2020-June/245226.html

[isanych]

@haubi, version which can be committed successfully is 3.1.4 from May 31st, version which causing issue is 3.1.4 from June 1st. You can install and use any version of cygwin, issue happens only when you trying to commit docker layer. It is not a cygwin bug, it's a docker bug and it is not cygwin specific, vs2015 has the same issue (and as cygwin it also was successfully installing before).

[maxried]

I'm also seeing this behaviour on Server 2019 with latest patches, and latest Docker installed. The workaround doesn't work for me (anymore?), because it seems as if the repository is gone.

[thaJeztah]

because it seems as if the repository is gone.

@maxried which repository is that?

[maxried]

@maxried which repository is that?

This cygwin site. It only returns a lot of 404s when I use it with cygwin, which then gives up. http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/2020/05/31/142136/

[isanych]

@maxried site still working for me, maybe it was temporary glitch.
I also created local mirror by uploading content of C:\tools\cygwin\http%3a%2f%2fctm.crouchingtigerhiddenfruitbat.org%2fpub%2fcygwin%2fcirca%2f64bit%2f2020%2f05%2f31%2f142136%2f and C:\tools\cygwin\cygwinsetup.exe to local web site, so now my images are working no matter what happening with ctm.crouchingtigerhiddenfruitbat.org

[voltamund]

I also had to investigate a failing docker build that uses the cygwin package from chocolatey. To me it looks like the reason why it fails with cygwin 3.1.5 and later is because the cygwin postinstall scripts now create WSL symlinks (IO_REPARSE_TAG_LX_SYMLINK) by default.
It looks like docker cannot handle these reparse points when the container is committed.

If I manually install the latest cygwin into an empty container and then try to commit the modified container into a new image, the commit fails with the described hcsshim::ImportLayer error. The commit is successful if I manually install cygwin 3.1.4 from the time machine http://www.crouchingtigerhiddenfruitbat.org. Note: The chocolatey package will always install the latest cygwin, so it is useless if your goal is to explicitly install an older cygwin version. I also found no way to specify the -X argument which seems to be required to use the time machine as install site.

After the installation of cygwin 3.1.4 the symlinks in dev will be created as plain files.

C:\cygwin\dev>dir /A
08/08/2020  01:50 AM    <DIR>          .
08/08/2020  01:50 AM    <DIR>          ..
08/08/2020  01:50 AM                40 fd
08/08/2020  01:50 AM    <DIR>          mqueue
08/08/2020  01:50 AM    <DIR>          shm
08/08/2020  01:50 AM                44 stderr
08/08/2020  01:50 AM                44 stdin
08/08/2020  01:50 AM                44 stdout

After installing cygwin 3.1.5 the same directory looks like this:

C:\cygwin64\dev>dir
 Directory of C:\cygwin64\dev
08/08/2020  09:33 AM    <DIR>          .
08/08/2020  09:33 AM    <DIR>          ..
08/08/2020  09:33 AM    <JUNCTION>     fd [...]
08/08/2020  09:32 AM    <DIR>          mqueue
08/08/2020  09:32 AM    <DIR>          shm
08/08/2020  09:33 AM    <JUNCTION>     stderr [...]
08/08/2020  09:33 AM    <JUNCTION>     stdin [...]
08/08/2020  09:33 AM    <JUNCTION>     stdout [...]

There are other directories which will contain symlinks.

The links are IO_REPARSE_TAG_LX_SYMLINK, e.g.

C:\cygwin64\dev>fsutil reparsePoint query fd
Reparse Tag Value : **0xa000001d**
Tag value: Microsoft
Tag value: Name Surrogate

Reparse Data Length: 0x11
Reparse Data:
0000:  02 00 00 00 2f 70 72 6f  63 2f 73 65 6c 66 2f 66  ..../proc/self/f
0010:  64                                                d

If I would delete all of the IO_REPARSE_TAG_LX_SYMLINK created by cygwin after the installation and then commit the container, it will succeed.

So far I have not found a way to tell cygwin to create symlinks using plain files during the installation, the CYGWIN environment variable seems to be ignored when running the postinstall scripts.

[lowenna]

@taylorb-microsoft and @jstarks The above is pretty good information and strongly implies an issue with the FS filter. Could one of you please forward this info to Scott and the relevant folks in the FS team? I don't believe there's anything that can be fixed here in OSS code. Thanks guys :)

[gdams]

Any update on this? I'm also getting the same issue

[wildmichael]

See the discussion on the Cygwin mailing list for a possible patch to setup.exe: http://cygwin.1069669.n5.nabble.com/Forcing-setup-exe-not-to-create-WSL-symlinks-tp153429p153478.html. Note that you will need to make sure that the CYGWIN environment variable is set to winsymlinks:lnk and that this value really reaches the post-install scripts. The problem with that is that Windows sanitizes the environment when it runs a process elevated through the runas verb which is how setup performs self-elevation. So, either the CYGWIN variable needs to be set for the machine or you make sure that setup doesn't perform an elevation step by setting CYGWIN in an already elevated shell/process and then passing the --no-admin (or -B) flag to setup.

[eembsen]

My colleague and I tested servercore:1809 with a variety of cygwin versions and installations and they all lead to the same ImportLayer/container commit problem. We tried to install version 3.0.0 to 3.1.5 and we tried to install directly using setup.exe and through chocolatey, nothing works! The only installation that works for us is the installation posted above with the site URL http://ctm.crouchingtigerhiddenfruitbat.org/pub/cygwin/circa/64bit/2020/05/31/142136/. So my question is, what is the difference between the cygwin packages on this site and the "official" cygwin packages?

[isanych]

@eembsen - these are official cygwin packages, just older version of them - site is a time machine for cygwin, but it run by enthusiast and may stop working at any moment, especially if traffic will grow significantly - so I suggest to create a local mirror by uploading content of C:\tools\cygwin\http%3a%2f%2fctm.crouchingtigerhiddenfruitbat.org%2fpub%2fcygwin%2fcirca%2f64bit%2f2020%2f05%2f31%2f142136%2f and C:\tools\cygwin\cygwinsetup.exe to local web site - that way you'll ensure that your images will continue to build even when site has issues and you'll reduce load on that site

[eembsen]

@isanych Thanks, will do that.
Is it just "older versions" of cygwin packages or are they compiled with special (GCC) flags. I would very much like to have a repeatable process. At the moment I feel a bit uncomfortable for the reason you mention.

[isanych]

just older versions - http://www.crouchingtigerhiddenfruitbat.org/Cygwin/timemachine.html#cygwintimemachine

[thecloudtaylor]

Currently the WCIFS filter (which is what implements the 'union' file system) does not support reparse points within the container layers that are not symlinks or junctions. All other reparse points (such as IO_REPARSE_TAG_LX_SYMLINK) are turned into IO_REPARSE_TAG_UNHANDLED.

This is the case because most other symlinks types are interpreted or handled by filesystem filters that don't run inside the container.

[paleozogt]

I've been unable to get the CYGWIN=winsymlinks:lnk solution to work when installing cygwin with choco in Docker.

However, I've figured out a workaround that, while gross and should make us all feel bad, does work.

The tl;dr is that you can change the symlinks with mklink so that they're compatible with Docker. There's not that many, so its actually tractable.

With a cygwin installation, you can tell which files are WSL symlinks like this:

C:\> choco install -y cygwin cyg-get
C:\> dir /S C:\tools\cygwin | C:\tools\cygwin\bin\grep JUNCTION
11/02/2020  02:11 PM    <JUNCTION>     fd [...]
11/02/2020  02:11 PM    <JUNCTION>     stderr [...]
11/02/2020  02:11 PM    <JUNCTION>     stdin [...]
11/02/2020  02:11 PM    <JUNCTION>     stdout [...]
11/02/2020  02:11 PM    <JUNCTION>     hosts [...]
11/02/2020  02:11 PM    <JUNCTION>     mtab [...]
11/02/2020  02:11 PM    <JUNCTION>     networks [...]
11/02/2020  02:11 PM    <JUNCTION>     protocols [...]
11/02/2020  02:11 PM    <JUNCTION>     services [...]
11/02/2020  02:11 PM    <JUNCTION>     bind.config [...]
11/02/2020  02:11 PM    <JUNCTION>     gnutls.config [...]
11/02/2020  02:11 PM    <JUNCTION>     java.config [...]
11/02/2020  02:11 PM    <JUNCTION>     krb5.config [...]
11/02/2020  02:11 PM    <JUNCTION>     libreswan.config [...]
11/02/2020  02:11 PM    <JUNCTION>     nss.config [...]
11/02/2020  02:11 PM    <JUNCTION>     openssh.config [...]
11/02/2020  02:11 PM    <JUNCTION>     opensshserver.config [...]
11/02/2020  02:11 PM    <JUNCTION>     openssl.config [...]
11/02/2020  02:11 PM    <JUNCTION>     opensslcnf.config [...]
11/02/2020  02:11 PM    <JUNCTION>     ca-bundle.legacy.crt [...]

Within cygwin bash, you can inspect the symlinks to find out how they're mapped. For example, for the files in /dev:

$ ls -l /dev/fd /dev/stdin /dev/stdout /dev/stderr
lrwxrwxrwx 1 IEUser None 13 Nov  2 14:11 /dev/fd -> /proc/self/fd
lrwxrwxrwx 1 IEUser None 15 Nov  2 14:11 /dev/stderr -> /proc/self/fd/2
lrwxrwxrwx 1 IEUser None 15 Nov  2 14:11 /dev/stdin -> /proc/self/fd/0
lrwxrwxrwx 1 IEUser None 15 Nov  2 14:11 /dev/stdout -> /proc/self/fd/1

This leads to the following workaround:

ARG CYGWIN_HOME=C:\tools\cygwin
RUN choco install --no-progress -y cygwin cyg-get && `
    cd %CYGWIN_HOME%\dev && `
    del fd && mklink fd "%CYGWIN_HOME%/proc/self/fd" && `
    del stdin && mklink stdin "%CYGWIN_HOME%/proc/self/fd/0" && `
    del stdout && mklink stdout "%CYGWIN_HOME%/proc/self/fd/1" && `
    del stderr && mklink stderr "%CYGWIN_HOME%/proc/self/fd/2" && `
    `
    cd %CYGWIN_HOME%\etc && `
    del hosts && mklink hosts "%WINDIR%\System32\drivers\etc\hosts" && `
    del mtab && mklink mtab "%CYGWIN_HOME%/proc/mounts" && `
    del networks && mklink networks "%WINDIR%\System32\drivers\etc\networks" && `
    del protocols && mklink protocols "%WINDIR%\System32\drivers\etc\protocol" && `
    del services && mklink services "%WINDIR%\System32\drivers\etc\services" && `
    `
    cd %CYGWIN_HOME%\etc\pki\ca-trust\source && `
    del ca-bundle.legacy.crt && mklink ca-bundle.legacy.crt "%CYGWIN_HOME%/usr/share/pki/ca-trust-legacy/ca-bundle.legacy.default.crt" && `
    `
    cd %CYGWIN_HOME%\etc\crypto-policies\back-ends && `
    del bind.config && mklink bind.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/bind.txt" && `
    del gnutls.config && mklink gnutls.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/gnutls.txt" && `
    del java.config && mklink java.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/java.txt" && `
    del krb5.config && mklink krb5.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/krb5.txt" && `
    del libreswan.config && mklink libreswan.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/libreswan.txt" && `
    del nss.config && mklink nss.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/nss.txt" && `
    del openssh.config && mklink openssh.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/openssh.txt" && `
    del opensshserver.config && mklink opensshserver.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/opensshserver.txt" && `
    del openssl.config && mklink openssl.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/openssl.txt" && `
    del opensslcnf.config && mklink opensslcnf.config "%CYGWIN_HOME%/usr/share/crypto-policies/DEFAULT/opensslcnf.txt"

Other packages that get installed will need similar workarounds, depending on the symlinks they use. For example, here's autoconf:

RUN cyg-get automake make && `
    cd %CYGWIN_HOME%\bin && `
    del autoconf && mklink autoconf "%CYGWIN_HOME%/usr/share/autotools/ac-wrapper.sh" && `
    del autoheader && mklink autoheader "%CYGWIN_HOME%/usr/share/autotools/ac-wrapper.sh" && `
    del autom4te && mklink autom4te "%CYGWIN_HOME%/usr/share/autotools/ac-wrapper.sh" && `
    del autoreconf && mklink autoreconf "%CYGWIN_HOME%/usr/share/autotools/ac-wrapper.sh" && `
    del autoscan && mklink autoscan "%CYGWIN_HOME%/usr/share/autotools/ac-wrapper.sh" && `
    del autoupdate && mklink autoupdate "%CYGWIN_HOME%/usr/share/autotools/ac-wrapper.sh" && `
    del ifnames && mklink ifnames "%CYGWIN_HOME%/usr/share/autotools/ac-wrapper.sh" && `
    cd %CYGWIN_HOME%\etc\alternatives && `
    del automake-doc && mklink automake-doc "%CYGWIN_HOME%/usr/share/info/automake1.9.info.gz"

I've put the code into a gist here.

[kempniu]

However, I've figured out a workaround that, while gross and should make us all feel bad, does work.

Thanks - I have a workaround which should make us all feel even worse! 🙃

RUN choco install -y cygwin & powershell -Command "Get-ChildItem -Path C:\tools -Recurse -Attributes ReparsePoint | Remove-Item"

Obviously this is horrible, but the symlinks do not seem to be necessary for running non-interactive Cygwin commands inside a Docker container. YMMV.

[LewisPringle]

I'm not sure who fixed what, but this problem has gone away for me.

This docker file https://github.com/SophistSolutions/Stroika/blob/d224e0/DockerBuildContainers/Windows-Cygwin-VS2k22/Dockerfile now builds for me.

Using the latest cygwin and choco to install. No promises, but hope this helps!