Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go tool dist test fail inside repo #26980

Closed
iWdGo opened this issue Aug 14, 2018 · 16 comments

Comments

Projects
None yet
3 participants
@iWdGo
Copy link
Contributor

commented Aug 14, 2018

When building Go, the run.bat part cannot be run correctly. It seems related to GOTOOLDIR which is incorrectly read or updated. On GOPATH, no issue occurs. Inside the repo, problems occur.
run.ba* checks that it is inside the repo (go\src).
After investigating, I suppose that this is why go tool dist cannot run inside the repo.

What version of Go are you using (go version)?

C:\Users{username is removed below}\Documents\Google>go version
go version go1.11rc1 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

C:\Users\{}\Documents\Google>go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\{}\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\{}\Documents\Google\
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\{}\AppData\Local\Temp\go-build523053410=/tmp/go-build -gno-record-gcc-switches

What did you do?

C:\Users\{}\Documents\Google>set GOTOOLDIR=C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

C:\Users\{}\Documents\Google>go env GOTOOLDIR
C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64

What did you see instead?

C:\Users\{}\Documents\Google>go env GOTOOLDIR
C:\Go\pkg\tool\windows_amd64

C:\Users\{}\Documents\Google>set GO
GOPATH=C:\Users\{}\Documents\Google\
GOROOT=C:\Go\
GOTOOLDIR=C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64

C:\Users\Costa\Documents\Google>set GOTOOLDIR=

C:\Users\Costa\Documents\Google>set GO
GOPATH=C:\Users\Costa\Documents\Google\
GOROOT=C:\Go\

C:\Users\Costa\Documents\Google>go env GOTOOLDIR
C:\Go\pkg\tool\windows_amd64

If you apply the same recipe to GOTMPDIR for instance, everything occurs as expected.
make.bat succeeds and displays the installation message. all.bat fails on the run.bat part with:

(lines removed)
Building packages and commands for windows/amd64.

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Users\\Costa\\Documents\\Google\\golang\\go\\go.exe": file does not exist

Starting run.bat with and w/o rebuild fails similarly:

C:\Users\Costa\Documents\Google\golang\go\src>run.bat --no-rebuild

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Go\\go.exe": file does not exist
C:\Users\{}\Documents\Google\golang\go\src>run.bat --no-rebuild --no-local

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Go\\go.exe": file does not exist
C:\Users\{}\Documents\Google\golang\go\src>set GO
GOARCH=amd64
GOBUILDFAIL=1
GOCACHE=C:\Users\{}\AppData\Local\go-build
GOEXE=.exe
GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\{}\AppData\Local\Temp\go-build370009818=/tmp/go-build -gno-record-gcc-switches
GOHOSTARCH=amd64
GOHOSTOS=windows
GOOS=windows
GOPATH=C:\Users\{username}\go
GOROOT=C:\Go
GOROOT_BOOTSTRAP=C:\Go\
GOTOOLDIR=C:\Go\pkg\tool\windows_amd64

After failure, GOPATH is set to a directory that does not exist.

I updated the base installation using rc1. The rest is run on tip.
I gave it a try but could not locate the GOTOOLDIR environment variable handling.

@agnivade

This comment has been minimized.

Copy link
Member

commented Aug 14, 2018

GOTOOLDIR is not meant to be updated.

Please see #26886 and #10264

@iWdGo iWdGo changed the title cmd/go: GOTOOLDIR cannot be updated cmd/go: go tool dist test fail inside repo Aug 14, 2018

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2018

Closing as duplicate of #26886.

Unlike many projects on GitHub, the Go project does not use its bug tracker for general discussion or asking questions. We only use our bug tracker for tracking bugs and tracking proposals going through the Proposal Process.

Please see https://golang.org/wiki/Questions for good places to ask questions.

@davecheney davecheney closed this Aug 14, 2018

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

Thanks for the confirmation. This is not explicit in the documentation of go env. It would be an improvement to confirm that GOTOOLDIR appears but cannot be changed to avoid any further issue.

The issue is then that run.bat fails and go tool dist test fails when run inside the repo. I updated the title accordingly. I still suspect that an invalid GOTOOLDIR is used or as underlined above the GOPATH is incorrectly set.

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

@davecheney I understand your point but go tool dist test still fails. I updated the title accordingly.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2018

It looks like the root failure is

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\Users\Costa\Documents\Google\golang\go\go.exe": file does not exist

This is not an expected error, and is not seen on our build dashboard or other users in the wild. Could some program you have installed on your computer, possibly an AV program, be interfering with the Go build process?

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

I build regularly and the issue started early last week. I re-installed Go with rc1. I clean where there are traces of the build. I have no alarm from AV and no error during build.
I confirm that out of the repo, the command succeeds. I can also run the build from inside go\src\bin.
I understand your argument but It might be related to Windows 10.
As you can see after failure, GOPATH is set to a non-existent directory.
I found this issue #18583 which is about the build process of Windows and revolves around GOPATH and USERPROFILE.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2018

@iWdGo building Go from source does not rely on GOPATH or being inside a repo.

Can you please post the complete output from your build invocation. Please do not update the issue description, that makes it hard for people to follow the flow of the issue as it evolves. Thank you.

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

Sorry for jumping the gun on the title. My focus is indeed this command inside the repo.
I re-started by computer as I can't remember when was the last time. The output of the command is identical.
GOPATH invalid value leads me to think that something occurs an intermediate environment variable.
The output of all.bat on tip this morning is as follows. I'm not sure that this is what you ask.

C:\Users\{}\Documents\Google\golang\go\src>go clean -cache

C:\Users\{}\Documents\Google\golang\go\src>time
The current time is:  9:03:17,91
Enter the new time:

C:\Users\{}\Documents\Google\golang\go\src>all.bat
Building Go cmd/dist using C:\Go\
Building Go toolchain1 using C:\Go\.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Users\\{}\\Documents\\Google\\golang\\go\\go.exe": file does not exist

C:\Users\{}\Documents\Google\golang\go\src>

C:\Users\{}\Documents\Google\golang\go\pkg\tool\windows_amd64 14-08-18 9:06
C:\Go\pkg\tool\windows_amd64 14-08-18 8:41

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2018

Thanks for your update. All I can suggest is something on your computer is blocking C:\Users\{}\Documents\Google\golang\go\go.exe" from being written. Not being a windows user myself I understand how frustrating it must be to be told "please turn off your AV", but this is often the cause of issues like this.

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

I believe that it is not trying to update to trying to execute because of the "exec:" word but I could not locate the related source code. There is never been a go.exe in the \go directory but only in the go\bin. And the installation has already occurred from what I can read. I confirm that make.bat finishes w/o error.
The structure of go directory is as expected:

14-08-18  09:11    <DIR>          .
14-08-18  09:11    <DIR>          ..
24-03-18  20:32               347 .gitattributes
30-03-18  21:01               270 .gitcookies
22-07-18  14:59    <DIR>          .github
24-03-18  20:32               928 .gitignore
24-07-18  12:59    <DIR>          api
13-08-18  16:10            55.284 AUTHORS
14-08-18  09:10    <DIR>          bin
24-03-18  20:32             1.339 CONTRIBUTING.md
13-08-18  16:10            71.070 CONTRIBUTORS
13-08-18  16:10    <DIR>          doc
24-03-18  20:32             5.686 favicon.ico
24-03-18  20:32    <DIR>          lib
24-03-18  20:32             1.479 LICENSE
22-07-18  14:54    <DIR>          misc
24-03-18  20:32             1.303 PATENTS
14-08-18  09:11    <DIR>          pkg
22-07-18  14:59             1.607 README.md
24-03-18  20:32                26 robots.txt
14-08-18  09:20    <DIR>          src
04-08-18  11:04    <DIR>          test
14-08-18  09:11                48 VERSION.cache

and go\bin:

 Directory of C:\Users\{}\Documents\Google\golang\go\bin

14-08-18  09:10    <DIR>          .
14-08-18  09:10    <DIR>          ..
14-08-18  09:13        12.742.656 go.exe
14-08-18  09:13         3.483.648 gofmt.exe

And from inside go\bin, everything occurs as expected:

C:\Users\Costa\Documents\Google\golang\go\bin>go version
go version devel +4fc7b93ad7 Mon Aug 13 19:04:16 2018 +0000 windows/amd64
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2018

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

A couple of details to confirm that the build works as go tool dist test -h and go tool dist -list both work as expected.
Here is the outcome of your suggestion:

C:\Users\Costa\Documents\Google\golang\go\src>set GOPATH=

C:\Users\Costa\Documents\Google\golang\go\src>set GOROOT=

C:\Users\Costa\Documents\Google\golang\go\src>
C:\Users\Costa\Documents\Google\golang\go\src>
C:\Users\Costa\Documents\Google\golang\go\src>go tool dist test
go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "C:\\Users\\Costa\\Documents\\Google\\golang\\go\\go.exe": file does not exist

C:\Users\Costa\Documents\Google\golang\go\src>set GO
Environment variable GO not defined

The last message confirms that there are no environment variables set which have a name starting with GO and I did the same for CGO. I add go env to confirm that values after failure. Thanks for keeping the issue open.

C:\Users\Costa\Documents\Google\golang\go\src>go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Costa\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\Costa\go
set GOPROXY=
set GORACE=
set GOROOT=C:\Users\Costa\Documents\Google\golang\go
set GOTMPDIR=
set GOTOOLDIR=C:\Users\Costa\Documents\Google\golang\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\Costa\AppData\Local\Temp\go-build408101810=/tmp/go-build -gno-record-gcc-switches

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2018

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2018

I'll keep searching anyway. Thanks for your replies. If anybody can point me to the relevant part of the code, it would be helpful as I already searched cmd/go in and out.

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2018

Here is a working solution. I post a complete explanation to detail the options. A key contradiction is that all.ba* (bash/bat) checks for the presence of make.bat. The understanding is that it must run from inside the repo used for build.

After build using make.bat , the build is usable from outside the repo which sits under the original GOPATH as documentation requires.

Create your workspace directory, %USERPROFILE%\go. (If you'd like to use a different directory, you will need to set the GOPATH environment variable.)

To illustrate the issue, here is what happen in various cases including what works. %USERPROFILE% is used everywhere and has been replaced manually:

C:\Users\Costa\Documents\Google\golang\go\src>set GO
GOPATH=%USERPROFILE%\Documents\Google\
GOROOT=C:\Go\
GOROOT_BOOTSTRAP=C:\Go\

GOPATH\golang\go\bin>go tool dist version
go1.11rc1

The version displayed in the GOROOT version.

Result here is as expected:

GOPATH\golang\go\bin>go version
go version devel +30d3ebe367 Tue Aug 21 09:05:51 2018 +0000 windows/amd64

GOPATH\golang\go>set GOROOT=%CD%

GOPATH\golang\go\src>set GO
GOPATH=%USERPROFILE%\Documents\Google\
GOROOT=%USERPROFILE%\Documents\Google\golang\go
GOROOT_BOOTSTRAP=C:\Go\

GOPATH\golang\go>cd src

GOPATH\golang\go\src>go tool dist version
devel +30d3ebe367 Tue Aug 21 09:05:51 2018 +0000

run.bat still fails

GOPATH\golang\go\src>run.bat --no-rebuild

go tool dist: FAILED: go install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exec: "%USERPROFILE%\\Documents\\Google\\golang\\go\\go.exe": file does not exist

set GOPATH=%USERPROFILE%\Download

C:\Users\Costa\Documents\Google\golang\go\src>set GO
GOPATH=%USERPROFILE%\Downloads
GOROOT=%USERPROFILE%\Documents\Google\golang\go
GOROOT_BOOTSTRAP=C:\Go\

Here, you can run go build:

%USERPROFILE%\Documents\Google\golang\go\src>cd %GOPATH%

%USERPROFILE%\Downloads>go tool dist version
devel +30d3ebe367 Tue Aug 21 09:05:51 2018 +0000

%USERPROFILE%\Downloads>go tool dist test -no-rebuild

##### Testing packages.
ok      archive/tar     2.088s
ok      archive/zip     4.410s

%USERPROFILE%\Downloads>go tool dist test -no-rebuild -run=^go_test:cmd/go$

##### Testing packages.
ok      cmd/go  380.826s

ALL TESTS PASSED (some were excluded)

It is feasible to fix the build, all.bat, run.bat or the documentation but I'm not certain on how it is supposed to work now or in the near future.

@iWdGo

This comment has been minimized.

Copy link
Contributor Author

commented Oct 21, 2018

Working on tip, issue is solved. Verbose flag must still be passed through GOFLAGS. Issue can be closed.

golang\go\src>go tool dist version
devel +035f9e8102 Sat Oct 20 14:19:29 2018 +0000
golang\go\src>set GOFLAGS=-v

golang\go\src>go tool dist test -no-rebuild -run=^go_test:encoding/xml$

##### Testing packages.
=== RUN   TestMarshal
=== RUN   TestMarshal/0
...

@agnivade agnivade closed this Oct 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.