cmd/go: running `go generate` with go tool with different GOROOT fails

[eliben]

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/eliben/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/eliben/eli/go"
GOPROXY="https://proxy.golang.org"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/eliben/eli/golang-go/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build618171788=/tmp/go-build -gno-record-gcc-switches"

What did you do?

1. Created a fresh clone of the Go repository.
2. $ cd src/cmd/compile/internal/gc
3. Edit syntax.go to add a new AST node type OFOOBAR

$ grep -C 4 OFOOBAR syntax.go 
// Node ops.
const (
	OXXX Op = iota

	OFOOBAR

	// names
	ONAME    // var or func name
ONONAME // unnamed arg or return value: f(int, string) (int, error) { etc }

4. Ran go generate

$ go generate 
stringer: writing output: open class_string.go: permission denied
/usr/local/go/src/cmd/compile/internal/gc/go.go:43: running "stringer": exit status 1

Expected op_string.go to be updated with the new node type.

Also ran:

$ go generate syntax.go

No errors, but the generated op_string.go does not have the new node type.

What did you expect to see?

Expected the new node type to appear in op_string.go after running go generate

[ianlancetaylor]

The key point here is that the go binary being run has a GOROOT that does not match the clone of the repo in which it is running. The current behavior when running in a clone is to run the generate command in the original GOROOT, as can be seen by editing the clone and the original GOROOT in two different ways. I don't know what the right thing is to do here, but silently warping over to a different directory and running generate there cannot be our best choice. We could run generate in the current directory, or we could give an error.

CC @bcmills @jayconrod

[bcmills]

I note in your go env output the line GOMOD="/home/eliben/eli/golang-go/src/go.mod".

That implies that you are working in a Go 1.13 GOROOT, but the go version output indicates that a Go 1.12 toolchain is being used. The Go 1.12 toolchain does not know how to work with the modules in a Go 1.13 GOROOT, so it's not surprising that this does not work.

What happens if you run go generate using a go toolchain built from a different GOROOT at head?

[eliben]

[I'm reproducing this on a different machine now, the original problem is still reproducible]

@bcmills

So I'm in ~/eli/golang-go/src/cmd/compile/internal/gc, where syntax.go has OFOOBAR added.

~/eli/golang-go/src/cmd/compile/internal/gc$ grep -C 4 OFOOBA syntax.go 
// Node ops.
const (
	OXXX Op = iota

	OFOOBAR

	// names
	ONAME    // var or func name
	ONONAME  // unnamed arg or return value: f(int, string) (int, error) { etc }

I built a new Go toolchain in a different directory, fresh from a clone taken a few minutes ago.

~/eli/golang-go/src/cmd/compile/internal/gc$ /usr/local/google/home/eliben/test/go-github/bin/go version
go version devel +41b9e99d5b Fri Jun 21 17:24:29 2019 +0000 linux/amd64

/usr/local/google/home/eliben/test/go-github/bin/go env Output

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/eliben/.cache/go-build"
GOENV="/usr/local/google/home/eliben/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/eliben/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/usr/lib/google-golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/usr/local/google/home/eliben/eli/golang-go/src/cmd/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build472372589=/tmp/go-build -gno-record-gcc-switches"

Running go generate with this toolchain, without modifying GOROOT:

~/eli/golang-go/src/cmd/compile/internal/gc$ /usr/local/google/home/eliben/test/go-github/bin/go generate syntax.go 
~/eli/golang-go/src/cmd/compile/internal/gc$ grep FOOBA op_string.go 

No luck.

[gertcuykens]

Trying to work on the src code of go doc but every time I try to compile a ./doc binary it uses other src code, it doesn't look in the current directory am in, I have to force it go build main.go

gert@gert ~/Desktop/go/src/cmd/doc:master> go build

gert@gert ~/Desktop/go/src/cmd/doc:master> ls
dirs.go         doc             doc_test.go     main.go         pkg.go          testdata

I can see a doc binary getting created but its not build form the main.go in current directory, I deliberately put a syntax error in main?

gert@gert ~/Desktop/go/src/cmd/doc:master> go build main.go 
# command-line-arguments
./main.go:78:2: syntax error: unexpected --, expecting }

details:

go version devel +f938b9b33b Wed Jun 26 20:26:48 2019 +0000 darwin/amd64

GOARCH="amd64"
GOBIN="/Users/gert/bin"
GOCACHE="/Users/gert/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/gert/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/gert/Desktop/go/src/cmd/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/dv/8tlwvjr91zjdyq4rk14lkkfm0000gn/T/go-build645754300=/tmp/go-build -gno-record-gcc-switches -fno-common"

Workaround for me was match GOROOT with the repo as in, git clone everything in /usr/local/go and bootstrap using a /usr/local/go1 then I could work on go doc src code.

[bcmills]

@gertcuykens, this issue is about go generate. It is not clear to me whether your issue has to do with go generate, but if it does not, please open a separate issue for investigation.

[gertcuykens]

@bcmills don't know for sure but Ian asked me to put it here :) see: https://groups.google.com/forum/#!topic/golang-nuts/j3LKg-Mel-s

[ianlancetaylor]

@bcmills I would expect that it's exactly the same problem: confusion between the current GOROOT and the toolchain's GOROOT. But we can open a separate issue for it if you prefer.

[bcmills]

It may well have the same root cause, but I prefer to have an issue for each symptom — that way I won't add a test for one symptom and forget to verify that it fixes the others.