Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "attempt to execute system stack code on user stack" during heap scavenging #34773

bcmills opened this issue Oct 8, 2019 · 7 comments


Copy link

@bcmills bcmills commented Oct 8, 2019

From the linux-amd64-nocgo builder (

fatal error: attempt to execute system stack code on user stack

goroutine 8 [running]:
runtime.throw(0x65ad83, 0x32)
	/workdir/go/src/runtime/panic.go:774 +0x72 fp=0xc000030610 sp=0xc0000305e0 pc=0x430562
	/workdir/go/src/runtime/stack.go:1315 +0x27 fp=0xc000030630 sp=0xc000030610 pc=0x44b117
runtime.stackfree(0xc000492000, 0xc000494000)
	/workdir/go/src/runtime/stack.go:422 +0x35c fp=0xc000030638 sp=0xc000030630 pc=0x448b2c
runtime.allocm(0xc00001e500, 0x65e5f8, 0x1)
	/workdir/go/src/runtime/proc.go:1517 +0xdb fp=0xc000030678 sp=0xc000030638 pc=0x43556b
runtime.newm(0x65e5f8, 0xc00001e500)
	/workdir/go/src/runtime/proc.go:1841 +0x39 fp=0xc0000306a8 sp=0xc000030678 pc=0x435d39
runtime.startm(0x0, 0xc000000c01)
	/workdir/go/src/runtime/proc.go:2006 +0x12a fp=0xc0000306d8 sp=0xc0000306a8 pc=0x43630a
runtime.ready(0xc000000c00, 0x0, 0xd5731d01)
	/workdir/go/src/runtime/proc.go:666 +0x12f fp=0xc000030728 sp=0xc0000306d8 pc=0x43359f
	/workdir/go/src/runtime/mgcscavenge.go:236 +0x85 fp=0xc000030750 sp=0xc000030728 pc=0x422d55
runtime.bgscavenge.func1(0x0, 0x0, 0x0)
	/workdir/go/src/runtime/mgcscavenge.go:295 +0x20 fp=0xc000030760 sp=0xc000030750 pc=0x45e6b0
	/workdir/go/src/runtime/time.go:294 +0x6e fp=0xc0000307d8 sp=0xc000030760 pc=0x44fabe
	/workdir/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc0000307e0 sp=0xc0000307d8 pc=0x463cd1
created by runtime.(*timersBucket).addtimerLocked
	/workdir/go/src/runtime/time.go:166 +0x10e

The failure message is the same as in #30591, but the entry here seems to be through runtime.wakeScavenger rather than runtime.LockOSThread so the root cause may differ.

CC @mknyszek @aclements

@bcmills bcmills added this to the Go1.14 milestone Oct 8, 2019
Copy link

@mknyszek mknyszek commented Oct 8, 2019

Looking into it.

@mknyszek mknyszek self-assigned this Oct 8, 2019
Copy link

@mknyszek mknyszek commented Oct 8, 2019

OK so ready() needs to be called on the system stack.

Either wakeScavenger should be go:systemstack or ready should be goready. I'm not sure whether it actually matters which one we do. No other part of wakeScavenger needs the system stack and I have a slight preference for allowing it to be run mostly anywhere, though in some cases (like SetGCPercent) this means we'll go onto the system stack twice. It doesn't happen in any place that's particularly performance sensitive.

@aclements, @randall77, WDYT?

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
Copy link

@randall77 randall77 commented Oct 14, 2019

I'd rather just put things on the system stack that need it. It serves as useful documentation, if nothing else. If that means we have to switch stacks twice for non-performance-sensitive code, so be it.

@randall77 randall77 modified the milestones: Backlog, Go1.14 Oct 14, 2019
Copy link

@gopherbot gopherbot commented Oct 14, 2019

Change mentions this issue: runtime: call goready in wakeScavenger instead of ready

@gopherbot gopherbot closed this in 2c87be4 Oct 15, 2019
Copy link

@lmm lmm commented Dec 12, 2019

@mknyszek @rsc do you know when this fix might appear in Go 1.13?

We're running into this occasionally in some integration tests. In what sorts of scenarios would this panic appear? (I don't know much about Golang GC). How does this panic get triggered?

Copy link

@aclements aclements commented Dec 13, 2019

@gopherbot, please open a backport to 1.13.

Copy link

@gopherbot gopherbot commented Dec 13, 2019

Backport issue(s) opened: #36127 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to

@golang golang locked and limited conversation to collaborators Dec 18, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.