Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: STW GC stops working #34736

cherrymui opened this issue Oct 7, 2019 · 3 comments

runtime: STW GC stops working #34736

cherrymui opened this issue Oct 7, 2019 · 3 comments


Copy link

@cherrymui cherrymui commented Oct 7, 2019

What version of Go are you using (go version)?

tip (cfe2320)

Does this issue reproduce with the latest release?

No. Go 1.13 works fine.

What operating system and processor architecture are you using (go env)?

darwin/amd64, linux/amd64

What did you do?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go

The compiler is just used as an example. This happens with essentially any non-trivial Go program. Setting GOGC=1 makes it happens more likely.

What did you expect to see?

Runs successfully.

What did you see instead?

$ GODEBUG=gcstoptheworld=1 go tool compile hello.go 
runtime: checkdead: find g 1 in status 1
fatal error: checkdead: runnable g

runtime stack:
runtime.throw(0x1a23d16, 0x15)
	/Users/cherryyz/src/go-tip/src/runtime/panic.go:774 +0x72
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:4273 +0x3a6
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:1963 +0x95
runtime.findrunnable(0xc000022500, 0x0)
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2428 +0x536
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2561 +0x2be
	/Users/cherryyz/src/go-tip/src/runtime/proc.go:2647 +0x9d
	/Users/cherryyz/src/go-tip/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [runnable]:
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/ssa.go:63 +0x308
	/Users/cherryyz/src/go-tip/src/cmd/compile/internal/gc/main.go:666 +0x2f94
	/Users/cherryyz/src/go-tip/src/cmd/compile/main.go:51 +0xac

cc @aclements

Copy link

@aclements aclements commented Oct 8, 2019

Some investigating;

cat >/tmp/hello.go <<EOF
package main
func main() { println("Hello") }
cat >/tmp/ <<EOF
set -e
cd src
for i in 1 2 3; do
    GODEBUG=gcstoptheworld=1 GOGC=1 go tool compile -o /tmp/x /tmp/hello.go
chmod a+x /tmp/
git bisect start master $(git merge-base master go1.13)
git bisect run /tmp/

And the result:

7b294cdd8df0a9523010f6ffc80c59e64578f34b is the first bad commit
commit 7b294cdd8df0a9523010f6ffc80c59e64578f34b
Author: Michael Anthony Knyszek <>
Date:   Mon Jun 17 19:03:09 2019 +0000

    runtime: don't hold worldsema across mark phase

cc @mknyszek

@mknyszek mknyszek added this to the Go1.14 milestone Oct 14, 2019
@mknyszek mknyszek self-assigned this Oct 14, 2019
Copy link

@mknyszek mknyszek commented Nov 22, 2019

I believe I found the problem, and it's actually the deadlock detector detecting a deadlock! Unfortunately the deadlock happens during a STW GC so checkdead observes runnable goroutines and panics.

The problem is that after 7b294cdd8 we no longer hold onto worldsema for the duration of a GC; we hold gcsema instead. However, in STW GC we don't release it before calling Gosched on the user goroutine, but because of how STW GC is implemented we will need to STW again before the GC is complete, and we'll be unable to because this disabled user goroutine is holding onto worldsema. The fix is easy, just drop worldsema before calling Gosched.

Copy link

@gopherbot gopherbot commented Nov 22, 2019

Change mentions this issue: runtime: release worldsema before Gosched in STW GC mode

@gopherbot gopherbot closed this in 05511a5 Nov 22, 2019
@golang golang locked and limited conversation to collaborators Nov 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants