cmd/go: build inside a Windows container does not find the sources in mounted volume

[StefanScherer]

What version of Go are you using (go version)?

go version go1.8beta2 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\gopath
set GORACE=
set GOROOT=c:\go
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

I tried to compile Go sources inside a Windows container by mounting the sources from the Windows Docker host into the Windows container.

On the Windows Docker host I did:

mkdir test
cd test
notepad webserver.go
docker run -it -v "$(pwd):C:\code" golang:1.8-nanoserver powershell

Now inside the PowerShell session in the Windows container I did:

cd c:\code
go build webserver.go

What did you expect to see?

It should find the Go source in the folder C:\code and build it.

I know there is #15978 with the fix e65bce7 in Go 1.8. That's why I tried it with the latest version.

What did you see instead?

PS C:\code> go build webserver.go
go build command-line-arguments: c:\go\pkg\tool\windows_amd64\compile.exe: chdir \ContainerMappedDirectories\F12A85AF-6803-4A43-A97B-BAFA93BDC334: The system cannot find the path specified.

Some more details about the mounted volume / reparse point C:\code in the container:

PS C:\> cmd /c dir
 Volume in drive C has no label.
 Volume Serial Number is 1A3C-1F59

 Directory of C:\

01/07/2017  01:35 AM    <SYMLINKD>     code [\\?\ContainerMappedDirectories\F12A85AF-6803-4A43-A97B-BAFA93BDC334]
01/07/2017  01:36 AM    <DIR>          go
01/07/2017  01:35 AM    <DIR>          gopath
11/20/2016  03:32 AM             1,894 License.txt
01/07/2017  01:35 AM    <DIR>          Program Files
07/16/2016  04:09 AM    <DIR>          Program Files (x86)
01/07/2017  01:35 AM    <DIR>          Users
01/07/2017  01:41 AM    <DIR>          Windows
               1 File(s)          1,894 bytes
               7 Dir(s)  21,231,828,992 bytes free
PS C:\> fsutil reparsepoint query c:\code
Reparse Tag Value : 0xa000000c
Tag value: Microsoft
Tag value: Name Surrogate
Tag value: Symbolic Link

Reparse Data Length: 0x00000116
Reparse Data:
0000:  00 00 80 00 82 00 86 00  00 00 00 00 5c 00 43 00  ............\.C.
0010:  6f 00 6e 00 74 00 61 00  69 00 6e 00 65 00 72 00  o.n.t.a.i.n.e.r.
0020:  4d 00 61 00 70 00 70 00  65 00 64 00 44 00 69 00  M.a.p.p.e.d.D.i.
0030:  72 00 65 00 63 00 74 00  6f 00 72 00 69 00 65 00  r.e.c.t.o.r.i.e.
0040:  73 00 5c 00 46 00 31 00  32 00 41 00 38 00 35 00  s.\.F.1.2.A.8.5.
0050:  41 00 46 00 2d 00 36 00  38 00 30 00 33 00 2d 00  A.F.-.6.8.0.3.-.
0060:  34 00 41 00 34 00 33 00  2d 00 41 00 39 00 37 00  4.A.4.3.-.A.9.7.
0070:  42 00 2d 00 42 00 41 00  46 00 41 00 39 00 33 00  B.-.B.A.F.A.9.3.
0080:  42 00 44 00 43 00 33 00  33 00 34 00 00 00 5c 00  B.D.C.3.3.4...\.
0090:  5c 00 3f 00 5c 00 43 00  6f 00 6e 00 74 00 61 00  \.?.\.C.o.n.t.a.
00a0:  69 00 6e 00 65 00 72 00  4d 00 61 00 70 00 70 00  i.n.e.r.M.a.p.p.
00b0:  65 00 64 00 44 00 69 00  72 00 65 00 63 00 74 00  e.d.D.i.r.e.c.t.
00c0:  6f 00 72 00 69 00 65 00  73 00 5c 00 46 00 31 00  o.r.i.e.s.\.F.1.
00d0:  32 00 41 00 38 00 35 00  41 00 46 00 2d 00 36 00  2.A.8.5.A.F.-.6.
00e0:  38 00 30 00 33 00 2d 00  34 00 41 00 34 00 33 00  8.0.3.-.4.A.4.3.
00f0:  2d 00 41 00 39 00 37 00  42 00 2d 00 42 00 41 00  -.A.9.7.B.-.B.A.
0100:  46 00 41 00 39 00 33 00  42 00 44 00 43 00 33 00  F.A.9.3.B.D.C.3.
0110:  33 00 34 00 00 00                                 3.4...
PS C:\> 

[alexbrainman]

This could be related to #17540.

Alex

[bradfitz]

We don't officially support Windows Nano Server yet, due to lack of Nano Server builders. That is #15287.

Maybe there's a commercial way to set them up yet. Nobody on our side has looked into that yet.

@jstarks, do you or your coworkers work on Docker for Windows? Any clues?

[StefanScherer]

A suitable build environment would be a Windows Server 2016 machine in GCE, Azure ...

If there is no template with Docker preinstalled, you can follow the docs to install Docker on a Windows Server 2016

Such a server can build Docker images based on these two base images:

• microsoft/windowsservercore
• microsoft/nanoserver

So if you want to have a look into Nano Server, using Docker would be easier than maintaining a Nano Server directly. But on Azure there also is a Nano Server VM available.

The issue with the symlink to \\?\ContainerMappedDirectories\uuid is also heavily discussed in moby/moby#27537 as it affects other languages as well.

[gopherbot]

CL https://golang.org/cl/41834 mentions this issue.

[alexbrainman]

@StefanScherer , please try https://go-review.googlesource.com/#/c/41834/
Does it fixes your problem?
Thank you.

Alex

[StefanScherer]

Thanks @alexbrainman, I already got the gopherbot notification and wondered if I can help test this. Is there any prebuilt binaries with that commit included so I can build a Windows image with it to try it out?

[StefanScherer]

At least Go 1.8.1 still has the problem

PS C:\code> go build webserver.go
go build command-line-arguments: c:\go\pkg\tool\windows_amd64\compile.exe: chdir \ContainerMappedDirectories\866206B9-824A-45EA-9A45-2A6B6F55BC65: The system cannot find the path specified.
PS C:\code> go version
go version go1.8.1 windows/amd64

[bradfitz]

@StefanScherer, the patches we're talking about here are for what will become Go 1.9. Go 1.8.1 is from the Go 1.8 branch which was frozen for new development about 6 months ago.

There isn't currently an easy way to get Windows binaries for tip. That is #18136.

[StefanScherer]

@bradfitz Thanks for the pointer. Yes using Go and finding issues and compiling it from source are some completely different things :-)

So getting started compiling easier would be a great benefit. Well I did that once for Docker and wrote a first Dockerfile for it as setting up a development machine was hard.

For Go it seems pretty easy.

FROM golang:1.8-windowsservercore
WORKDIR /compile
RUN git clone https://github.com/golang/go
WORKDIR /compile/go/src
ENV GOROOT_BOOTSTRAP C:/go
ENV CGO_ENABLED 0
RUN cmd /C all.bat

OK, some test errors are there, probably not all the tests run well inside a container as well. But I have two binaries:

$ docker run golang-builder '..\bin\go.exe' version
go version devel +60db9fb6bc Fri Apr 28 18:34:49 2017 +0000 windows/amd64

Just a short feedback if compiling it this way could do the job for testing the latest master branch would be fine.

[bradfitz]

If you just want the binaries, just change all.bat to make.bat in your Dockerfile and skip the tests.

[StefanScherer]

Thanks. So I tried this binary by appending another line to the Dockerfile

RUN cp ..\bin\*.exe \go\bin

and then running a container with a mounted volume from the host:

$ docker run -it -v C:$(pwd):C:/code golang-builder powershell

Windows PowerShell 
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\compile\go\src> go version
go version devel +60db9fb6bc Fri Apr 28 18:34:49 2017 +0000 windows/amd64

PS C:\code> go build webserver.go
can't load package: package main: cannot find package "." in:
C:\code

PS C:\code> cd ..
PS C:\> mkdir xxx
PS C:\> cd xxx
PS C:\xxx> copy \code\webserver.go .
PS C:\xxx> go build webserver.go
PS C:\xxx> dir


    Directory: C:\xxx


Mode                LastWriteTime         Length Name                                                                                  
----                -------------         ------ ----                                                                                  
-a----        4/28/2017  12:48 PM        6362112 webserver.exe                                                                         
-a----        3/10/2017   2:55 PM            270 webserver.go                                                                          

So there is another error message with this new go.exe.
Can I turn on some debugging to show you more info?

[StefanScherer]

I have created a gist with the output of the build to show the failed tests, but they seem to me unrelated.
https://gist.github.com/StefanScherer/ef596640cee1c37519bb9a5a437b44a0

Only this

--- FAIL: TestNetworkSymbolicLink (0.01s)
	os_windows_test.go:482: The Server service is not started.

might be interesting.

[StefanScherer]

I've added the Dockerfile to my dockerfiles-windows repo and let AppVeyor run the same build and test: https://ci.appveyor.com/project/StefanScherer/dockerfiles-windows/build/1.0.469
Now it's easy for me to run a re-build and test it again and again.

[alexbrainman]

Sorry it is bit long. I hope you can translate it into Docker file or something. I don't know much about how that works.

If you want to build current Go tip (or any other version of Go) on clean Windows computer you need:

1. git;

2. another Go distro pre-built by someone else - this is to bootstrap your build process.

I suspect you already have git installed.

So you just need #2. For #2 you can use any Go version go1.4 and above. It is important to note that #2 binaries should not be listed in your PATH and #2 should not be pointed by GOROOT environment variable. It is GOROOT_BOOTSTRAP should be pointing to the root of #2.

The simplest way to get your hands on #2 is to download it from https://storage.googleapis.com/golang/go1.8.1.windows-amd64.zip. I would not use .msi file, because it adjust your PATH and some GO variables. I have downloaded that file and extracted it into C:\a\go1.8.1.

Now you can do download current Go tip:

C:\a>git clone https://go.googlesource.com/go
Cloning into 'go'...
remote: Sending approximately 148.40 MiB ...
remote: Counting objects: 7197, done
remote: Finding sources: 100% (24/24)
remote: Total 302808 (delta 237915), reused 302798 (delta 237915)
Receiving objects: 100% (302808/302808), 147.90 MiB | 1.62 MiB/s, done.
Resolving deltas: 100% (237915/237915), done.
Checking connectivity... done.
Checking out files: 100% (6545/6545), done.

C:\a>dir
 Volume in drive C has no label.
 Volume Serial Number is F0C8-6DFE

 Directory of C:\a

29/04/2017  03:24 PM    <DIR>          .
29/04/2017  03:24 PM    <DIR>          ..
29/04/2017  03:27 PM    <DIR>          go
07/04/2017  05:01 PM    <DIR>          go1.8.1
               0 File(s)              0 bytes
               4 Dir(s)  79,904,595,968 bytes free

C:\a>

Next thing I always do is create small batch file, like this gocmd.bat:

set MYHOME=c:\a
set GOROOT=%MYHOME%\go
set GOROOT_BOOTSTRAP=%MYHOME%\go1.8.1
set GOPATH=%MYHOME%
set CGO_ENABLED=0
set PATH=%PATH%;%GOROOT%\bin
cd %GOROOT%\src
CMD

This file setup all environment variables for you to be able use and rebuild Go tip. I just copy this file to my desktop, I click on it and I get command window I could use to develop.

set GOROOT=%MYHOME%\go
this tells where my Go repo is. Some people say, you should not set this, but I do. Perhaps I am wrong. But I have been doing this for too long, and that works for me - as long as you don't make mistake.

set GOROOT_BOOTSTRAP=%MYHOME%\go1.8.1
this should be set on your pre-built version you've downloaded before.

set GOPATH=%MYHOME%
this is not important for your exercise, but generally just set it and forget it.

set CGO_ENABLED=0
you have to set CGO_ENABLED to 0, because you don't have gcc installed.

set PATH=%PATH%;%GOROOT%\bin
adds c:\a\go\bin directory to your PATH - this way new go.exe will be found when you type commands in new command window.

cd %GOROOT%\src
goes to the c:\a\go\src directory (my favourite place, because I work inside Go repo), but you can go somewhere else, for example %GOPATH%\src also nice sometimes.

CMD
starts command window.

Well you click on the file, and once command line opens you should run make.bat to build Go version you have checked out, like:

...
c:\a\go\src>make
##### Building Go bootstrap tool.
cmd/dist

##### Building Go toolchain using c:\a\go1.8.1.
bootstrap/cmd/internal/objabi
bootstrap/cmd/internal/dwarf
...
...
...
cmd/vet/internal/cfg
cmd/vet/internal/whitelist
cmd/vet


---
Installed Go for windows/amd64 in c:\a\go
Installed commands in c:\a\go\bin

c:\a\go\src>

Once Go is built you can click gocmd.bat again and again, and use your go.exe without running make.bat again. You only need to run it again if you want to checkout different commit and rebuild against new commit. For example to test CL 41834, you would have to run:

git fetch https://go.googlesource.com/go refs/changes/34/41834/3 && git checkout FETCH_HEAD

and then run make.bat will give you go.exe and all libs as for CL 41834 Patch Set 3.

You can do all this exercise outside of Docker, and then pass c:\a into Docker with your "$(pwd):C:\code" trick. That is what I did. I had to copy my gocmd.bat into docker_gocmd.bat and modify MYHOME variable. I also had to set PATH inside docker directly (without using %PATH% trick), because your golang:1.8-nanoserver already adds some Go version to the PATH. You don't want two different go.exe version in your PATH.

You can also run make.bat inside Docker. You can also use already installed Go version (from golang:1.8-nanoserver) for GOROOT_BOOTSTRAP. You only have to remember to modify your PATH so that old go.exe is not listed there - you only want go.exe from directory where you're building to be in the PATH.

OK, some test errors are there, probably not all the tests run well inside a container as well.

What Brad said. Do not call all.bat. all.bat builds Go + runs all tests. If people don't develops Go inside Docker, then noone report bugs. So there might be some bugs. Even non-Go bugs :-) Just run make.bat to build Go, and test what you care about.

RUN cp ..\bin*.exe \go\bin

Looks like you copy go.exe (and others) from one Go tree into the other. Do not do that. make.bat builds your *.exe and *.o and then you use them. You can copy whole tree, but do not mix and match *.exe and *.o from different source trees. It is also important that sources match *.exe and *.o - otherwise go will start rebuilding them automatically to match them.

can't load package: package main: cannot find package "." in:

I said do not do "RUN cp ..\bin*.exe \go\bin", but it somehow worked. This is the error message I saw on current tip. But if you apply CL 41834, you should be able to build your program.

--- FAIL: TestNetworkSymbolicLink (0.01s)
os_windows_test.go:482: The Server service is not started.

The TestNetworkSymbolicLink uses Server service. And that is not running inside your Docker. Surprisingly, we didn't have anyone complain about this yet. Do not run all.bat.

Hope it helps.

Alex

[StefanScherer]

Thank you @alexbrainman for the detailed steps. I've updated my Dockerfile with commit StefanScherer/dockerfiles-windows@195a969, first move away the original /go 1.8.1 inside the base image, then clone and fetch the CL 41834, then build it at /go/src to have the final binaries in PATH /go/bin.

With that image I can build in a folder mounted from my Docker host (even from my Mac that hosts my windows-docker-machine VM):

$ docker run -it -v C:$(pwd):C:/code golang-issue-18555 powershell

Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\go\src> cd ..
PS C:\go> cd ..
PS C:\> cd code
PS C:\code> dir


    Directory: C:\code


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
------        4/13/2017   3:24 PM             69 build.ps1
------        4/13/2017   4:27 PM            210 Dockerfile
------        3/26/2017  10:24 AM            220 push.ps1
------        3/26/2017  10:24 AM           1344 README.md
------        3/26/2017  10:24 AM             99 test.ps1
------        3/10/2017   2:55 PM            270 webserver.go


PS C:\code> go build webserver.go
PS C:\code> dir


    Directory: C:\code


Mode                LastWriteTime         Length Name                          
----                -------------         ------ ----                          
------        4/13/2017   3:24 PM             69 build.ps1                     
------        4/13/2017   4:27 PM            210 Dockerfile                    
------        3/26/2017  10:24 AM            220 push.ps1                      
------        3/26/2017  10:24 AM           1344 README.md                     
------        3/26/2017  10:24 AM             99 test.ps1                      
------        4/30/2017   9:05 AM        6339584 webserver.exe                 
------        3/10/2017   2:55 PM            270 webserver.go                  


PS C:\code> 

The AppVeyor build also shows a "green" build and test, a compiled webserver.exe can be found on the host.

So the CL 41834 solves the problem. 👍

[alexbrainman]

So the CL 41834 solves the problem

Thank you for checking.
I will wait for someone to review CL 41834 so it can get submitted.

Alex