cmd/compile: compiler built with PGO crashes occasionally on ppc64{le}

[prattmic]

#!watchflakes
default <- goarch ~ `ppc64` && date != "" && date > "2023-05-22" && date < "2023-06-01"

https://go.dev/cl/495596 added a default.pgo profile for cmd/compile, enabling a PGO build of the compiler (as long as -pgo=none is not set explicitly).

This caused a variety of crashes in the compiler on ppc64{le} builders:

2023-05-18T16:55:07-88f89d8/linux-ppc64le-power9osu
2023-05-18T13:41:27-33a601b/aix-ppc64
2023-05-18T12:52:14-7b0835d/aix-ppc64
2023-05-18T10:23:17-75add1c/aix-ppc64
2023-05-18T09:16:07-774f602/linux-ppc64-sid-power10
2023-05-18T09:16:07-774f602/linux-ppc64le-buildlet
2023-05-18T09:15:25-27906bb/aix-ppc64
2023-05-18T09:15:25-27906bb/linux-ppc64-sid-power10
2023-05-18T01:40:37-6ed8474/linux-ppc64-sid-buildlet
2023-05-18T01:40:37-6ed8474/linux-ppc64-sid-power10
2023-05-18T00:35:53-956d31e/linux-ppc64le-buildlet
2023-05-17T22:11:31-0b86a04/linux-ppc64le-buildlet
2023-05-17T21:53:11-c426c87/linux-ppc64-sid-buildlet
2023-05-17T21:53:11-c426c87/linux-ppc64le-power10osu
2023-05-17T21:44:30-2693ade/linux-ppc64-sid-power10

That CL also caused #60263. Since it caused several issues, the CL was reverted in https://go.dev/cl/496185. #60263 has since been fixed.

Given these failures are all on ppc64{le}, @dr2chase and I suspect that they are due to a bad ppc64-specific optimization (SSA rule, e.g.) that is tickled by the additional inlining caused by PGO.

I have had some success reproducing these crashes. Running all.bash in a loop on three linux-ppc64-sid-power10 builders concurrently with GOGC=5 usually gets me a failure in <30 minutes. Not stellar, but I think workable.

We should then be able to bisect down to a bad function with GOSSAHASH applied in inlineCostOK to enable/disable PGO-based inlining. (Also set -d=pgodevirtualize=0 to disable PGO-based devirtualization, which was submitted in https://go.dev/cl/492436, after the bad CL above was reverted).

Given that there is a path forward to debugging ppc64, and we'd like more soak time on the primary ports, I intend to resubmit https://go.dev/cl/495596, which will make ppc64 flaky until this issue is resolved. (GOARCH=ppc64{le} could also temporarily change the default of -pgo from auto to none if necessary).

cc @golang/ppc64 @dr2chase @aclements @cherrymui

[gopherbot]

Change https://go.dev/cl/497455 mentions this issue: cmd/compile: build compiler with PGO

[prattmic]

My current repro approach:

$ export GOROOT=/my/dev/tree/
$ export GOMOTE_GROUP=ppc64_pgo
$ gomote create -count=3 -setup linux-ppc64-sid-power10
$ gomote run -until=goroutine -e GOGC=5 ./go/bin/go build -a cmd/dist

i.e., rebuild the cmd/dist over and over until we get a crash ("goroutine" should appear in stack trace of any compiler crash).

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- goarch ~ `ppc64` && date != "" && date > "2023-05-22"

2023-05-23 18:15 linux-ppc64le-buildlet go@b0f15b4a cmd/cgo/internal/testshared.TestTrivialExecutable (log)

--- FAIL: TestTrivialExecutable (3.68s)
    shared_test.go:491: executing /workdir/go/bin/go install -linkshared ./trivial failed exit status 1:
        # runtime
        panic: runtime error: index out of range [7156992] with length 7898 [recovered]
        	panic: runtime error: index out of range [7156992] with length 7898

        goroutine 1 [running]:
        panic({0x928e40?, 0xc0026367e0?})
        	runtime/panic.go:1017 +0x448 fp=0xc0022ec640 sp=0xc0022ec580 pc=0x58a78
        runtime.goPanicIndex(0x6d3500, 0x1eda)
...
        cmd/compile/internal/gc.compileFunctions()
        	cmd/compile/internal/gc/compile.go:194 +0x220 fp=0xc0022ed968 sp=0xc0022ed8e8 pc=0x8595b0
        cmd/compile/internal/gc.Main(0x999188)
        	cmd/compile/internal/gc/main.go:344 +0x1764 fp=0xc0022edea8 sp=0xc0022ed968 pc=0x85b904
        main.main()
        	cmd/compile/main.go:57 +0x134 fp=0xc0022edf18 sp=0xc0022edea8 pc=0x8807b4
        runtime.main()
        	runtime/proc.go:267 +0x348 fp=0xc0022edfc0 sp=0xc0022edf18 pc=0x5ba88
        runtime.goexit()
        	runtime/asm_ppc64x.s:912 +0x4 fp=0xc0022edfc0 sp=0xc0022edfc0 pc=0x94a94

— watchflakes

[laboger]

We have done some experimenting with this. I found that if we build the default.pgo file natively instead of using the one built on amd64 we are not able to get the problem to occur. When using the default pgo file built on amd64 more inlining happens within cmd/compile/internal/ssagen.(*state).stmt and according to the stacktraces that seems like where many of the problems occur.

I was going to attempt to force more aggressive inlining and see if that forces the problem.

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- goarch ~ `ppc64` && date != "" && date > "2023-05-22"

2023-05-24 01:03 linux-ppc64le-power10osu go@bdc5533f cmd/compile/internal/ssa [build] (log)

panic: runtime error: index out of range [7177776] with length 32378

goroutine 346 [running]:
cmd/internal/src.(*PosTable).Pos(...)
	cmd/internal/src/xpos.go:153
cmd/internal/obj.(*Link).InnermostPos(...)
	cmd/internal/obj/inl.go:124
cmd/internal/obj.(*Link).OutermostPos(0x10?, {0x80000?, 0xc0?})
	cmd/internal/obj/inl.go:105 +0x134
cmd/compile/internal/base.FmtPos({0x11?, 0x0?})
...
	cmd/compile/internal/ssagen/ssa.go:548 +0x235c
cmd/compile/internal/ssagen.Compile(0xc01047f340, 0x0?)
	cmd/compile/internal/ssagen/pgen.go:187 +0x48
cmd/compile/internal/gc.compileFunctions.func5.1(0x0?)
	cmd/compile/internal/gc/compile.go:184 +0x4c
cmd/compile/internal/gc.compileFunctions.func3.1()
	cmd/compile/internal/gc/compile.go:166 +0x48
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 90
	cmd/compile/internal/gc/compile.go:165 +0x20c
go tool dist: FAILED: /workdir/go/pkg/tool/linux_ppc64le/go_bootstrap install -a cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- goarch ~ `ppc64` && date != "" && date > "2023-05-22"

2023-05-24 03:36 linux-ppc64le-buildlet go@f0d575c2 cmd/cgo/internal/testshared.TestTrivialExecutable (log)

--- FAIL: TestTrivialExecutable (4.81s)
    shared_test.go:491: executing /workdir/go/bin/go install -linkshared ./trivial failed exit status 1:
        # runtime
        panic: runtime error: index out of range [50463491] with length 7908 [recovered]
        	panic: runtime error: index out of range [50463491] with length 7908

        goroutine 1 [running]:
        panic({0x929740?, 0xc004ff4ee8?})
        	runtime/panic.go:1017 +0x448 fp=0xc007406640 sp=0xc007406580 pc=0x58d88
        runtime.goPanicIndex(0x3020303, 0x1ee4)
...
        cmd/compile/internal/gc.compileFunctions()
        	cmd/compile/internal/gc/compile.go:194 +0x220 fp=0xc007407968 sp=0xc0074078e8 pc=0x85ae80
        cmd/compile/internal/gc.Main(0x999ec8)
        	cmd/compile/internal/gc/main.go:344 +0x1760 fp=0xc007407ea8 sp=0xc007407968 pc=0x85d1d0
        main.main()
        	cmd/compile/main.go:57 +0x134 fp=0xc007407f18 sp=0xc007407ea8 pc=0x882ae4
        runtime.main()
        	runtime/proc.go:267 +0x348 fp=0xc007407fc0 sp=0xc007407f18 pc=0x5bd98
        runtime.goexit()
        	runtime/asm_ppc64x.s:912 +0x4 fp=0xc007407fc0 sp=0xc007407fc0 pc=0x94ee4

— watchflakes