cmd/compile: "internal compiler error: … unhandled expr" on darwin-amd64-10_11 builder

[bcmills]

2020-02-22T15:25:30-638df87/darwin-amd64-10_11

It's not obvious to me whether this is Darwin filesystem flakiness or a regression introduced in the recent cmd/compile refactoring. We should determine which is the case prior to the 1.15 release.

# cmd/compile/internal/ssa
src/cmd/compile/internal/ssa/rewritegeneric.go:14999:22: internal compiler error: 'rewriteValuegeneric_OpNeq8': unhandled expr =

goroutine 9 [running]:
runtime/debug.Stack(0x1a07de0, 0xc0000ac008, 0x0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/runtime/debug/stack.go:24 +0x9d
cmd/compile/internal/gc.Fatalf(0xc00e370ba0, 0x17, 0xc001b70360, 0x2, 0x2)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/subr.go:193 +0x291
cmd/compile/internal/gc.(*ssafn).Fatalf(0xc02241b6e0, 0x3a971600000003c, 0x189e0f6, 0x11, 0xc0220e36c0, 0x1, 0x1)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:6821 +0x1b0
cmd/compile/internal/gc.(*state).Fatalf(...)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:662
cmd/compile/internal/gc.(*state).expr(0xc022412d80, 0xc00f179b80, 0x0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:2722 +0x11ac
cmd/compile/internal/gc.(*state).storeArgWithBase(0xc022412d80, 0xc00f179b80, 0xc001296ae0, 0xc00dd4d8e8, 0x0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:5048 +0xd9
cmd/compile/internal/gc.(*state).storeArg(0xc022412d80, 0xc00f179b80, 0xc001296ae0, 0x0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:5029 +0x52
cmd/compile/internal/gc.(*state).call(0xc022412d80, 0xc006d8d100, 0xc00dd5de00, 0x0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:4462 +0x101a
cmd/compile/internal/gc.(*state).stmt(0xc022412d80, 0xc006d8d100)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1055 +0xda1
cmd/compile/internal/gc.(*state).stmtList(0xc022412d80, 0xc00072d1c0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1019 +0x58
cmd/compile/internal/gc.(*state).stmt(0xc022412d80, 0xc0038b1880)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1395 +0x25d2
cmd/compile/internal/gc.(*state).stmtList(0xc022412d80, 0xc00072d2c0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1019 +0x58
cmd/compile/internal/gc.(*state).stmt(0xc022412d80, 0xc006985100)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1395 +0x25d2
cmd/compile/internal/gc.(*state).stmtList(0xc022412d80, 0xc00072d4c0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1019 +0x58
cmd/compile/internal/gc.(*state).stmt(0xc022412d80, 0xc00399bf80)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1395 +0x25d2
cmd/compile/internal/gc.(*state).stmtList(0xc022412d80, 0xc00072dfa0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:1019 +0x58
cmd/compile/internal/gc.buildssa(0xc00a1f3b80, 0x1, 0x0)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/ssa.go:427 +0xc39
cmd/compile/internal/gc.compileSSA(0xc00a1f3b80, 0x1)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc0100ee480, 0xc0100d4b50, 0x1)
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
	/private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

go tool dist: FAILED: /private/var/folders/dx/k53rs1s93538b4x20g46cj_w0000gn/T/workdir-host-darwin-10_11/go/pkg/tool/darwin_amd64/go_bootstrap install -gcflags=all= -ldflags=all= -a -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2

CC @randall77 @josharian

[josharian]

That smells an awful lot like memory corruption. Note that the compiler is deterministic, so if any compiler changes caused this, it should happen every time, so it is pretty unlikely to be my compiler changes. (In theory this could be caused by a race, but the race-enabled compiler builder hasn't complained, I don't see how my changes could have introduced a race, and we have two failures from the same builder, which suggests hardware.)

I tried reproducing via gomote, running make.bash in a loop, to try to bisect just in case, but I don't have any failures so far. However, there are three different builders, and if it is a hardware problem, I might not have lucked into getting the problematic hardware. Is there any way to select a particular builder as a gomote?

[bcmills]

Hmm, #23011 is probably relevant. Looks like we've declared an intent to drop support for macOS 10.11.

@dmitshur, @toothrot, @cagedmantis: given that the 10.11 builder seems to be flaky, would it make sense to go ahead and disable it for the main branch?

[dmitshur]

@bcmills I'm not sure I understand the suggestion. Both Go 1.14 and 1.13 still support macOS 10.11, so it'll be another 12 months before we no longer support macOS 10.11 at all (when Go 1.15 becomes the oldest supported Go release).

[bcmills]

@dmitshur, we should adjust the buildsRepo functions so that the builder is not active for configurations that do not support it.

(See https://go-review.googlesource.com/c/build/+/169498/6/dashboard/builders.go#1862 for an example.)

[dmitshur]

Ah, I missed the "for the main branch" bit. Yes, we should do that, since Go 1.15 won't support macOS 10.11.

[dmitshur]

We've looked at this release-blocking issue in a release meeting today.

It's not obvious to me whether this is Darwin filesystem flakiness or a regression introduced in the recent cmd/compile refactoring. We should determine which is the case prior to the 1.15 release.

Given that the macOS 10.11 builder is disabled on master branch (see discussion above and #37425), we're not actively getting signal from it, so I don't think we can use it to determine if there is a regression in cmd/compile.

Recent builds on supported macOS builders are passing on master, release-branch.go1.14, and release-branch.go1.13 branches from a look at build.golang.org now (with some flaky failures, but in places other than in cmd/compile).

I can't think of anything more we can do for this issue other than to close it as resolved. Do you agree @bcmills, or do you have another suggestion for next steps here?

[dmitshur]

I can't think of anything more we can do for this issue other than to close it as resolved. Do you agree @bcmills, or do you have another suggestion for next steps here?

Friendly ping @bcmills on this.

[bcmills]

Yep, I agree that this is resolved.