runtime: "fatal: systemstack called from unexpected goroutine" on Android

[bcmills]

greplogs --dashboard -md -l -e '^fatal: systemstack called from unexpected goroutine' --since=2021-01-01

2022-02-02T21:12:39-53d6a72/android-amd64-emu

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
fatal: systemstack called from unexpected goroutineTrap 
exitcode=133
FAIL	runtime	59.879s

2021-10-08T16:26:20-59d4e92-99c1b24/android-amd64-emu

fatal: systemstack called from unexpected goroutineSegmentation fault 
exitcode=139FAIL	golang.org/x/net/publicsuffix	3.419s

I'll also note that badsystemstackMsg seems to be missing a final newline as of CL 93659 (CC @aclements @randall77). 😅

[bcmills]

This happened in a TryBot in https://storage.googleapis.com/go-build-log/f1e11825/android-amd64-emu_262486a5.log:

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
fatal: systemstack called from unexpected goroutineTrap 
exitcode=133
FAIL	runtime	16.177s
FAIL
2022/05/03 14:18:09 Failed: exit status 1
go tool dist: FAILED

Marking as release-blocker because this affects TryBot runs. Since android/amd64 is not a first-class port, either the underlying bug can be diagnosed and fixed, or the builder can be removed from the default TryBot set. (I'll leave that choice up to @golang/runtime to decide and implement.)

[bcmills]

This may or may not be OS-specific. There is another failure in the builder logs since February, but on plan9 rather than android; it isn't obvious to me whether that is an independent bug.

greplogs -l -e 'fatal: systemstack called from unexpected goroutine' --since=2022-02-03
2022-03-05T21:20:16-e155b03-45f4544/plan9-amd64-0intro

[bcmills]

greplogs -l -e 'fatal: systemstack called from unexpected goroutine' --since=2022-03-06
2022-05-03T19:48:07-bccce90/android-arm64-corellium

[mknyszek]

@golang/runtime This is a second class port, but because it's a trybot, this is a release blocker. Should we consider removing this as a trybot? Is that bringing us enough value?

[gopherbot]

Change https://go.dev/cl/407615 mentions this issue: dashboard: remove android-amd64-emu from main go repo's TryBot set

[dmitshur]

I've mailed CL 407615 that makes android-amd64-emu a post-submit builder only (in the main repo) while investigation of this issue is underway. If submitted, this issue can be unmarked as a release-blocker for Go 1.19.

[bcmills]

Curiously, this does not appear to be arch-specific: we've seen these failures on both amd64 and arm64.

greplogs -l -e 'fatal: systemstack called from unexpected goroutine' --since=2022-05-04
2022-05-20T22:30:37-2b0e457/android-arm64-corellium