cmd/compile: cmd/compile/internal/gc.buildssa panic

[howardjohn]

What version of Go are you using (go version)?

$ go version
go version go1.14.2 linux/amd64

Does this issue reproduce with the latest release?

Yes, this was done on 1.14.2

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/gobin"
GOCACHE="/gocache"
GOENV="/home/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build048816231=/tmp/go-build -gno-record-gcc-switches"

This was built inside a docker image: gcr.io/istio-testing/build-tools:master-2020-04-10T20-55-56.

What did you do?

Tried to build our application.

What did you expect to see?

Build succeeds

What did you see instead?

STATIC=0 GOOS=windows LDFLAGS='-extldflags -static -s -w' common/scripts/gobuild.sh /tmp/tmp.W9NXZv7L5L/build/work/src/istio.io/istio/out/linux_amd64/release/istioctl-win.exe ./istioctl/cmd/istioctl
go: downloading github.com/inconshreveable/mousetrap v1.0.0
# k8s.io/apimachinery/pkg/util/runtime
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x74 pc=0xce6f4f]
goroutine 8 [running]:
cmd/compile/internal/gc.buildssa(0x0, 0x3, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:297 +0xdf
cmd/compile/internal/gc.compileSSA(0x0, 0x3)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc0000ef4a0, 0xc000018e10, 0x3)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

Full CI log: https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/22958/release-test_istio/11141
Rerunning the job with the same code the build succeeds: https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/22958/release-test_istio/11153

The referenced gobuild.sh script is https://github.com/istio/istio/blob/master/common/scripts/gobuild.sh.

I think the fact its in the compile code path means its a golang bug and not an issue with our code, but maybe not. Let me know if more info is needed.

[randall77]

Definitely looks like a bug in our compiler. The pointer to the function to be compiled is nil.

Is this reproducible? Does it happen every time, or is it intermittent?

Did this happen in earlier versions of Go? (1.4.{0,1}, and 1.13.10 would be good ones to try if you haven't yet.)

I'll try to reproduce this myself.

[randall77]

How do you populate ./istioctl/cmd/istioctl? I don't see anything in the log that makes that directory tree.

[howardjohn]

@randall77 I have only see this a single time in our CI, and rerunning on the exact same job it did not occur. So its intermittent, I am not sure if its occurred more than once since we don't have a way to query through all of our test logs and I have only seen it once.

We upgrading our CI to go 1.14.2 yesterday and previously were using 1.13.10, so its possible its related to that change - I'll update this if we see this again.

The CI job is calling this make target: https://github.com/istio/istio/blob/e33092b5c59d1c13fb11d86ab6345f45194133d3/Makefile.core.mk#L403 which calls https://github.com/istio/istio/blob/e33092b5c59d1c13fb11d86ab6345f45194133d3/Makefile.core.mk#L385 which calls gobuild.sh ./istioctl/cmd/istioctl which calls go build ./istioctl/cmd/istioctl with some flags

[randall77]

Sorry, I was asking about the source code. Presumably there are some .go files in an istioctl/cmd/istioctl/ directory somewhere. Where can I get that?

If this is intermittent, it is going to be hard to debug. There aren't any obvious ways a nil pointer might make its way into a the compile queue.

[cuonglm]

@randall77 Maybe this's a sign for #33485

[howardjohn]

Ah I see. That would be https://github.com/istio/istio/tree/master/istioctl/cmd/istioctl. I'll keep an eye out for any re-occurrences as well, totally understand this isn't a lot of info to go on.

[randall77]

Two more examples from #41077

2020-08-27T18:33:37-ac2a1f8/linux-amd64-sid
2020-08-14T17:45:39-a20cb4c/linux-amd64-nocgo

[mdempsky]

Based on my comments in #33485, I wonder if it's related to the call to dtypesym in gc/ssa.go? In particular, could calling dtypesym be causing functions to get pushed onto compilequeue, which would be bad since that slice is being concurrently accessed by the goroutine responsible for distributing functions to the worker goroutines?

Edit: Maybe not. There's already a check for Siggen just before the call to dtypesym.

[toothrot]

@randall77 @mdempsky Checking in on this as a 1.16 release blocker. Has anybody managed to reproduce this yet?