Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: internal compiler error while building targets for kubernetes with master go #44370

Closed
Rajalakshmi-Girish opened this issue Feb 18, 2021 · 17 comments
Assignees
Milestone

Comments

@Rajalakshmi-Girish
Copy link

@Rajalakshmi-Girish Rajalakshmi-Girish commented Feb 18, 2021

What version of Go are you using (go version)?

$ go version

go version devel +f0be3cc547 Thu Feb 18 04:30:23 2021 +0000 linux/ppc64le
go version devel +f0be3cc547 Thu Feb 18 04:30:23 2021 +0000 linux/amd64

Observing this from the commit ID 2f0da6d9e2

What operating system and processor architecture are you using (go env)?

Operating sytem: Linux
Architecture: ppc64le and amd64

What did you do?

Building targets for kubernetes using the latest golang in master branch

What did you expect to see?

Kubernetes binaries build without error

What did you see instead?

Below compilation error:

[root@rajalakshmi-workspace kubernetes]# KUBE_BUILD_PLATFORMS=linux/ppc64le make cross
+++ [0218 08:42:30] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
Generating prerelease lifecycle code for 24 targets
+++ [0218 08:42:35] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
Generating deepcopy code for 226 targets
+++ [0218 08:42:46] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
Generating defaulter code for 88 targets
+++ [0218 08:42:59] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
Generating conversion code for 123 targets
+++ [0218 08:43:25] Building go targets for linux/ppc64le:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
Generating openapi code for KUBE
Generating openapi code for AGGREGATOR
Generating openapi code for APIEXTENSIONS
Generating openapi code for CODEGEN
Generating openapi code for SAMPLEAPISERVER
+++ [0218 08:43:42] Building go targets for linux/ppc64le:
    ./vendor/github.com/go-bindata/go-bindata/go-bindata
+++ [0218 08:43:44] Building go targets for linux/ppc64le:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/kube-scheduler
    vendor/k8s.io/kube-aggregator
    vendor/k8s.io/apiextensions-apiserver
    cluster/gce/gci/mounter
+++ [0218 08:49:18] Building go targets for linux/ppc64le:
    cmd/kube-proxy
    cmd/kubeadm
    cmd/kubelet
+++ [0218 08:49:44] Building go targets for linux/ppc64le:
    cmd/kubectl
+++ [0218 08:50:05] Building go targets for linux/ppc64le:
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/genswaggertypedocs
    cmd/linkcheck
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    cluster/images/conformance/go-runner
# k8s.io/kubernetes/test/e2e/common
test/e2e/common/util.go:208:30: internal compiler error: 'rcByNamePort.func1': addr of canSSA expression:
.   NAME-rc.i tc(1) Class:PPARAM Offset:0 OnStack Used int32 # rc_utils.go:55

goroutine 271 [running]:
runtime/debug.Stack(0x9dd688, 0xc00000e018, 0x0)
        /root/go/src/runtime/debug/stack.go:24 +0x8c
cmd/compile/internal/base.FatalfAt(0x372c0000003e0, 0xc005faa990, 0x24, 0xc0038c35a0, 0x2, 0x2)
        /root/go/src/cmd/compile/internal/base/print.go:227 +0x19c
cmd/compile/internal/base.Fatalf(...)
        /root/go/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/ssagen.(*ssafn).Fatalf(0xc0064a4060, 0x372c0000003e0, 0x8a54b7, 0x1e, 0xc006448890, 0x1, 0x1)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:7210 +0x160
cmd/compile/internal/ssagen.(*state).Fatalf(...)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:845
cmd/compile/internal/ssagen.(*state).addr(0xc00649e400, 0x9ecfe0, 0xc003281ee0, 0x0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:5000 +0x130
cmd/compile/internal/ssagen.(*state).expr(0xc00649e400, 0x9eb550, 0xc003282ff0, 0x0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:2860 +0x4664
cmd/compile/internal/ssagen.(*state).stmt(0xc00649e400, 0x9eb870, 0xc004655e00)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1498 +0x27c
cmd/compile/internal/ssagen.(*state).stmtList(0xc00649e400, 0xc0036c1ee0, 0x1, 0x1)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1269 +0x6c
cmd/compile/internal/ssagen.(*state).stmt(0xc00649e400, 0x9ed490, 0xc003282fa0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1584 +0x3c08
cmd/compile/internal/ssagen.(*state).stmtList(0xc00649e400, 0xc0036c1eb0, 0x1, 0x1)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1269 +0x6c
cmd/compile/internal/ssagen.buildssa(0xc00325c840, 0x0, 0x0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:522 +0x974
cmd/compile/internal/ssagen.Compile(0xc00325c840, 0x0)
        /root/go/src/cmd/compile/internal/ssagen/pgen.go:168 +0x44
cmd/compile/internal/gc.compileFunctions.func2.1(0xc00465a000, 0xc00325c840, 0xc003e9ffa0, 0xc0007ca1d0)
        /root/go/src/cmd/compile/internal/gc/compile.go:127 +0x50
created by cmd/compile/internal/gc.compileFunctions.func2
        /root/go/src/cmd/compile/internal/gc/compile.go:125 +0x80

# k8s.io/kubernetes/test/e2e/network
test/e2e/network/service.go:1660:34: internal compiler error: 'glob..func24.21.7': addr of canSSA expression:
.   NAME-rc.i tc(1) Class:PPARAM Offset:0 OnStack Used int32 # rc_utils.go:55

goroutine 324 [running]:
runtime/debug.Stack(0x9dd688, 0xc00000e018, 0x0)
        /root/go/src/runtime/debug/stack.go:24 +0x8c
cmd/compile/internal/base.FatalfAt(0x372c000000b15, 0xc009ab5020, 0x24, 0xc006da5e80, 0x2, 0x2)
        /root/go/src/cmd/compile/internal/base/print.go:227 +0x19c
cmd/compile/internal/base.Fatalf(...)
        /root/go/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/ssagen.(*ssafn).Fatalf(0xc009c97c80, 0x372c000000b15, 0x8a54b7, 0x1e, 0xc009ca85a0, 0x1, 0x1)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:7210 +0x160
cmd/compile/internal/ssagen.(*state).Fatalf(...)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:845
cmd/compile/internal/ssagen.(*state).addr(0xc009cba500, 0x9ecfe0, 0xc006a169c0, 0x0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:5000 +0x130
cmd/compile/internal/ssagen.(*state).expr(0xc009cba500, 0x9eb550, 0xc006a12b40, 0x0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:2860 +0x4664
cmd/compile/internal/ssagen.(*state).stmt(0xc009cba500, 0x9eb870, 0xc007b8af50)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1498 +0x27c
cmd/compile/internal/ssagen.(*state).stmtList(0xc009cba500, 0xc0072325f0, 0x1, 0x1)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1269 +0x6c
cmd/compile/internal/ssagen.(*state).stmt(0xc009cba500, 0x9ed490, 0xc006a12af0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1584 +0x3c08
cmd/compile/internal/ssagen.(*state).stmtList(0xc009cba500, 0xc0072325a0, 0x1, 0x1)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:1269 +0x6c
cmd/compile/internal/ssagen.buildssa(0xc0069fc6e0, 0x1, 0x0)
        /root/go/src/cmd/compile/internal/ssagen/ssa.go:522 +0x974
cmd/compile/internal/ssagen.Compile(0xc0069fc6e0, 0x1)
        /root/go/src/cmd/compile/internal/ssagen/pgen.go:168 +0x44
cmd/compile/internal/gc.compileFunctions.func2.1(0xc0070c2780, 0xc0069fc6e0, 0xc007b45380, 0xc0004d5510)
        /root/go/src/cmd/compile/internal/gc/compile.go:127 +0x50
created by cmd/compile/internal/gc.compileFunctions.func2
        /root/go/src/cmd/compile/internal/gc/compile.go:125 +0x80

!!! [0218 08:53:13] Call tree:
!!! [0218 08:53:13]  1: /root/kubernetes/hack/lib/golang.sh:874 kube::golang::build_binaries_for_platform(...)
!!! [0218 08:53:13]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0218 08:53:13] Call tree:
!!! [0218 08:53:13]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0218 08:53:13] Call tree:
!!! [0218 08:53:13]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make[1]: *** [Makefile:92: all] Error 1
make: *** [Makefile:508: cross] Error 1
[root@rajalakshmi-workspace kubernetes]#```
@Rajalakshmi-Girish
Copy link
Author

@Rajalakshmi-Girish Rajalakshmi-Girish commented Feb 18, 2021

@laboger ^ ^

@ALTree ALTree changed the title `internal compiler error` while building targets for kubernetes with master go cmd/compile: internal compiler error while building targets for kubernetes with master go Feb 18, 2021
@ALTree ALTree added this to the Go1.17 milestone Feb 18, 2021
@cuonglm
Copy link
Contributor

@cuonglm cuonglm commented Feb 18, 2021

The relevant line which causes problem https://github.com/kubernetes/kubernetes/blob/960e5e78255dd148d4dae49f62e729ea940f4f07/test/e2e/framework/rc/rc_utils.go#L55

Seems another instance of inlining OCLOSURE.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Feb 18, 2021

@johejo
Copy link

@johejo johejo commented Feb 18, 2021

I found a failure similar.

go version

version devel +609d82b289 Thu Feb 18 15:00:59 2021 +0000 linux/amd64

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/heijo/.cache/go-build"
GOENV="/home/heijo/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/heijo/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/heijo/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/heijo/ghq/go.googlesource.com/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/heijo/ghq/go.googlesource.com/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel +609d82b289 Thu Feb 18 15:00:59 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/heijo/ghq/github.com/cilium/cilium/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2455150930=/tmp/go-build -gno-record-gcc-switches"
git clone https://github.com/cilium/cilium.git
cd cilium/
go build ./...
# github.com/cilium/cilium/pkg/service/store
pkg/service/store/store.go:189:35: internal compiler error: 'JoinClusterServices.func2': addr of canSSA expression:
.   NAME-lock.i tc(1) Class:PAUTO Offset:0 OnStack Used int64 # stoppable_waitgroup.go:38

goroutine 59 [running]:
runtime/debug.Stack(0xe9d600, 0xc0001aa008, 0x0)
        /home/heijo/ghq/go.googlesource.com/go/src/runtime/debug/stack.go:24 +0x9f
cmd/compile/internal/base.FatalfAt(0x2633000000044, 0xc00001cc60, 0x24, 0xc00015cd80, 0x2, 0x2)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/base/print.go:227 +0x1bc
cmd/compile/internal/base.Fatalf(...)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/base/print.go:196
cmd/compile/internal/ssagen.(*ssafn).Fatalf(0xc000164390, 0x2633000000044, 0xd762bf, 0x1e, 0xc0007f5a70, 0x1, 0x1)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:7210 +0x1a7
cmd/compile/internal/ssagen.(*state).Fatalf(...)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:845
cmd/compile/internal/ssagen.(*state).addr(0xc000137400, 0xeacdc0, 0xc00049dd40, 0x0)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:5000 +0x171
cmd/compile/internal/ssagen.(*state).expr(0xc000137400, 0xeab330, 0xc000445900, 0x0)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:2860 +0x5334
cmd/compile/internal/ssagen.(*state).stmt(0xc000137400, 0xeab650, 0xc00080d680)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:1498 +0x2ae
cmd/compile/internal/ssagen.(*state).stmtList(0xc000137400, 0xc0007f5270, 0x1, 0x1)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:1269 +0x68
cmd/compile/internal/ssagen.(*state).stmt(0xc000137400, 0xead270, 0xc0004458b0)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:1584 +0x42de
cmd/compile/internal/ssagen.(*state).stmtList(0xc000137400, 0xc00080e480, 0x4, 0x4)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:1269 +0x68
cmd/compile/internal/ssagen.buildssa(0xc00043ab00, 0x3, 0x0)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/ssa.go:522 +0x1145
cmd/compile/internal/ssagen.Compile(0xc00043ab00, 0x3)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/ssagen/pgen.go:168 +0x5d
cmd/compile/internal/gc.compileFunctions.func2.1(0xc0000b2180, 0xc00043ab00, 0xc0001ab000, 0xc000126960)
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/gc/compile.go:127 +0x65
created by cmd/compile/internal/gc.compileFunctions.func2
        /home/heijo/ghq/go.googlesource.com/go/src/cmd/compile/internal/gc/compile.go:125 +0x8e
@danscales danscales self-assigned this Feb 18, 2021
@danscales
Copy link

@danscales danscales commented Feb 23, 2021

OK, neither of the fixes for #44325 or #44335 fix this bug, so I will take a look at it in detail over the next few days.

@laboger
Copy link
Contributor

@laboger laboger commented Feb 23, 2021

I think it is very likely this is due to the recent change to trampolines in https://go-review.googlesource.com/c/go/+/292490, since Kubernetes is large enough to require trampolines for long distance calls.

@laboger
Copy link
Contributor

@laboger laboger commented Feb 23, 2021

@pmur FYI

@tosi3k
Copy link

@tosi3k tosi3k commented Feb 24, 2021

FYI: we have a CI job building Kubernetes using Golang tip version running every 4 hours: https://testgrid.k8s.io/sig-scalability-golang#build-and-push-k8s-at-golang-tip. These builds are later used to run load tests to detect performance regressions in Kubernetes coming from Golang.

Last successful build job was using this CL as the tip: https://go-review.googlesource.com/c/go/+/275297, so something that was introduced later than the aforementioned "trampolines" change.

@aclements
Copy link
Member

@aclements aclements commented Feb 24, 2021

Quick repro with k8s (no make required):

git clone https://github.com/kubernetes/kubernetes
cd kubernetes/test/e2e/common
# Checkout current master, just so we're on the same page
git checkout 6235ad8c74f7
go build
@gopherbot
Copy link

@gopherbot gopherbot commented Feb 24, 2021

Change https://golang.org/cl/296049 mentions this issue: cmd/compile: disable inlining functions with closures for now

gopherbot pushed a commit that referenced this issue Feb 24, 2021
Added a flag '-d=inlfuncswithclosures=1' to allow inlining functions with
closures, and change the default to off for now, until #44370 is fixed.

Updates #44370.

Change-Id: Ic17723aa5c091d91f5f5004d8b63ec7125257acf
Reviewed-on: https://go-review.googlesource.com/c/go/+/296049
Run-TryBot: Dan Scales <danscales@google.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Dan Scales <danscales@google.com>
@danscales
Copy link

@danscales danscales commented Feb 24, 2021

OK, if you sync to change 8027343 or beyond on master (tip), I disabled inlining functions with closures, so this bug should no longer happen. Meanwhile, I will work on figure out the problem and checking in the real fix.

@OneOfOne
Copy link
Contributor

@OneOfOne OneOfOne commented Feb 24, 2021

@danscales I feel like making it disabled by default because of one project is wrong, they could have easily disabled it for their builds.

@randall77
Copy link
Contributor

@randall77 randall77 commented Feb 24, 2021

@OneOfOne I disagree. Just because they are the first to notice, doesn't mean this bug won't affect lots of others. Including introducing possible random trybot failures for everyone else.
If the fix isn't immediately apparent, always roll back correctness bugs, fix them in a non-time-sensitive manner, and then roll forward.

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Feb 25, 2021

@OneOfOne Inlining of functions with closures is a new feature that hasn't been released yet, and we intentionally enabled it by default knowing it was still WIP to help smoke out issues like this. Now that we have a handful of concrete, reproducible issues to use for testing, it makes sense to turn it back off by default while we fix those issues.

A few months worth of changes have been merged recently from dev.regabi and dev.typeparams, and they're suddenly getting much broader test exposure. Like @randall77 points out, it helps to rollback individual changes/features once we know they have issues so they don't hide further issues in other changes or ongoing development.

@OneOfOne
Copy link
Contributor

@OneOfOne OneOfOne commented Feb 25, 2021

Fair enough, for whatever reason I thought this was enabled since 1.15 and the regabi merge did something.

Carry on, I appreciate all that work.

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 25, 2021

Change https://golang.org/cl/296649 mentions this issue: cmd/compile: fix missing descend in Addrtaken for closures.

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Feb 25, 2021

Incidentally, it looks like CL 286434 (7eaaf28) is already helping prevent silent miscompilations by turning them into noisy ICEs. :)

@gopherbot gopherbot closed this in 9a555fc Feb 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet