runtime: STW GC stops working

[cherrymui]

What version of Go are you using (go version)?

tip (cfe2320)

Does this issue reproduce with the latest release?

No. Go 1.13 works fine.

What operating system and processor architecture are you using (go env)?

darwin/amd64, linux/amd64

What did you do?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go

The compiler is just used as an example. This happens with essentially any non-trivial Go program. Setting GOGC=1 makes it happens more likely.

What did you expect to see?

Runs successfully.

What did you see instead?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go 
runtime: checkdead: find g 1 in status 1
fatal error: checkdead: runnable g

runtime stack:
runtime.throw(0x1a23d16, 0x15)
	/Users/cherryyz/src/go-tip/src/runtime/panic.go:774 +0x72
runtime.checkdead()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:4273 +0x3a6
runtime.mput(...)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:4664
runtime.stopm()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:1963 +0x95
runtime.findrunnable(0xc000022500, 0x0)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2428 +0x536
runtime.schedule()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2561 +0x2be
runtime.park_m(0xc00047a000)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2647 +0x9d
runtime.mcall(0x80000)
	/Users/cherryyz/src/go-tip/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [runnable]:
cmd/compile/internal/gc.initssaconfig()
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/ssa.go:63 +0x308
cmd/compile/internal/gc.Main(0x1a3baa8)
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/main.go:666 +0x2f94
main.main()
	/Users/cherryyz/src/go-tip/src/cmd/compile/main.go:51 +0xac

cc @aclements

[aclements]

Some investigating;

cat >/tmp/hello.go <<EOF
package main
func main() { println("Hello") }
EOF
cat >/tmp/bisect.sh <<EOF
set -e
cd src
./make.bash
for i in 1 2 3; do
    GODEBUG=gcstoptheworld=1 GOGC=1 go tool compile -o /tmp/x /tmp/hello.go
done
EOF
chmod a+x /tmp/bisect.sh
git bisect start master $(git merge-base master go1.13)
git bisect run /tmp/bisect.sh

And the result:

7b294cdd8df0a9523010f6ffc80c59e64578f34b is the first bad commit
commit 7b294cdd8df0a9523010f6ffc80c59e64578f34b
Author: Michael Anthony Knyszek <mknyszek@google.com>
Date:   Mon Jun 17 19:03:09 2019 +0000

    runtime: don't hold worldsema across mark phase

cc @mknyszek