runtime: STW GC stops working

[cherrymui]

What version of Go are you using (go version)?

tip (cfe2320)

Does this issue reproduce with the latest release?

No. Go 1.13 works fine.

What operating system and processor architecture are you using (go env)?

darwin/amd64, linux/amd64

What did you do?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go

The compiler is just used as an example. This happens with essentially any non-trivial Go program. Setting GOGC=1 makes it happens more likely.

What did you expect to see?

Runs successfully.

What did you see instead?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go 
runtime: checkdead: find g 1 in status 1
fatal error: checkdead: runnable g

runtime stack:
runtime.throw(0x1a23d16, 0x15)
	/Users/cherryyz/src/go-tip/src/runtime/panic.go:774 +0x72
runtime.checkdead()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:4273 +0x3a6
runtime.mput(...)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:4664
runtime.stopm()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:1963 +0x95
runtime.findrunnable(0xc000022500, 0x0)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2428 +0x536
runtime.schedule()
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2561 +0x2be
runtime.park_m(0xc00047a000)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2647 +0x9d
runtime.mcall(0x80000)
	/Users/cherryyz/src/go-tip/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [runnable]:
cmd/compile/internal/gc.initssaconfig()
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/ssa.go:63 +0x308
cmd/compile/internal/gc.Main(0x1a3baa8)
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/main.go:666 +0x2f94
main.main()
	/Users/cherryyz/src/go-tip/src/cmd/compile/main.go:51 +0xac

cc @aclements

[aclements]

Some investigating;

cat >/tmp/hello.go <<EOF
package main
func main() { println("Hello") }
EOF
cat >/tmp/bisect.sh <<EOF
set -e
cd src
./make.bash
for i in 1 2 3; do
    GODEBUG=gcstoptheworld=1 GOGC=1 go tool compile -o /tmp/x /tmp/hello.go
done
EOF
chmod a+x /tmp/bisect.sh
git bisect start master $(git merge-base master go1.13)
git bisect run /tmp/bisect.sh

And the result:

7b294cdd8df0a9523010f6ffc80c59e64578f34b is the first bad commit
commit 7b294cdd8df0a9523010f6ffc80c59e64578f34b
Author: Michael Anthony Knyszek <mknyszek@google.com>
Date:   Mon Jun 17 19:03:09 2019 +0000

    runtime: don't hold worldsema across mark phase

cc @mknyszek

[mknyszek]

I believe I found the problem, and it's actually the deadlock detector detecting a deadlock! Unfortunately the deadlock happens during a STW GC so checkdead observes runnable goroutines and panics.

The problem is that after 7b294cdd8 we no longer hold onto worldsema for the duration of a GC; we hold gcsema instead. However, in STW GC we don't release it before calling Gosched on the user goroutine, but because of how STW GC is implemented we will need to STW again before the GC is complete, and we'll be unable to because this disabled user goroutine is holding onto worldsema. The fix is easy, just drop worldsema before calling Gosched.

[gopherbot]

Change https://golang.org/cl/208379 mentions this issue: runtime: release worldsema before Gosched in STW GC mode