runtime: race between stack shrinking and channel send/recv leads to bad sudog values

[mknyszek]

Internally we've seen a rare crash arise in the runtime since Go 1.14. The error message is typically sudog with non-nil elem stemming from a call to releaseSudog from chansend or chanrecv.

The issue here is a race between a mark worker and a channel operation. Consider the following sequence of events. GW is a worker G. GS is a G trying to send on a channel. GR is a G trying to receive on that same channel.

1. GW wants to suspend GS to scan its stack. It calls suspendG.
2. GS is about to gopark in chansend. It calls into gopark, and changes its status to _Gwaiting BEFORE calling its unlockf, which sets gp.activeStackChans.
3. GW observes _Gwaiting and returns from suspendG. It continues into scanstack where it checks if it's safe to shrink the stack. In this case, it's fine. So, it reads gp.activeStackChans, and sees it as false. It begins adjusting sudog pointers without synchronization. It reads the sudog's elem pointer from the chansend, but has not written it back yet.
4. GS continues on its merry way and sets gp.activeStackChans and parks. It doesn't really matter when this happens at this point.
5. GR comes in and wants to chanrecv on channel. It grabs the channel lock, reads from the sudog's elem field, and clears it. GR readies GS.
6. GW then writes the updated sudog's elem pointer and continues on its merry way.
7. Sometime later, GS wakes up because it was readied by GR, and tries to release the sudog, which has a non-nil elem field.

The fix here, I believe, is to set gp.activeStackChans before the unlockf is called. Doing this ensures that the value is updated before any worker that could shrink GS's stack observes a useful G status in suspendG. This could alternatively be fixed by changing the G status after unlockf is called, but I worry that will break a lot of things.

CC @aclements @prattmic

[mknyszek]

This should be fixed for Go 1.14 and Go 1.15. It's a bug that was introduced in Go 1.14, and may cause random and unavoidable crashes at any point in time. There may not be enough time to fix this for 1.15 (the failure is very rare, but we've seen it internally), and if not, it should definitely go in a point release.

@gopherbot please open a backport issue for 1.14.

[gopherbot]

Backport issue(s) opened: #40642 (for 1.14), #40643 (for 1.15).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/247050 mentions this issue: runtime: set activeStackChans before gopark

[mknyszek]

Oh no! That fix doesn't work because if there's a stack growth at any point in between, we could self-deadlock. Ugh. OK I don't know what the right fix is yet...

[cherrymui]

Could we set GS as not safe for shrink stack?

[mknyszek]

Yeah, that might work. I'm not sure if it's technically safe for stack scanning though because if it's a chanrecv falling into this path then there could be a write into the stack in that same window from a sender? I guess that's already true (or we handle this case) today?

[cherrymui]

I think stack scanning should be fine, as it doesn't write to the stack. It is okay to read the old or the new value. As long as the write has a write barrier, it should be fine.

[mknyszek]

Thanks Cherry, that makes sense. sendDirect has an explicit write barrier and we're not worried about reads of a partially-written pointer in stack scanning because memmove guarantees it'll copy a pointer-word a time.

Your solution would mean that we could probably get rid of syncadjustsudogs altogether which would be a nice simplification! 😀 A stack shrink will still be enqueued for the next sync preemption so it's probably also fine from a memory use perspective.

[prattmic]

If we set it as not safe for stack shrinking, that means we can't shrink stacks of anything blocked on chan send, right? Would that be too limiting, given that such G's may be blocked indefinitely? I certainly regularly see thousands of G's blocked in chan recv, though I imagine chan send is less common.

[mknyszek]

If we set it as not safe for stack shrinking, that means we can't shrink stacks of anything blocked on chan send, right? Would that be too limiting, given that such G's may be blocked indefinitely? I certainly regularly see thousands of G's blocked in chan recv, though I imagine chan send is less common.

It applies to both chan send and chan recv in this case. Maybe it is too limiting? I hadn't considered the case where Gs are blocked indefinitely and you want to shrink their stacks. If the G will run soon, then the next synchronous preemption will likely shrink its stack (unless it's another channel op...).