runtime: speed up GC phase transitions #81657

go version go1.28-devel_d27b23f671 Wed Sep 16 18:07:03 2026 -0700 darwin/arm64

I've been looking at how the GC coordinates across all Ps, for the GC's ragged barriers and STWs. (Most of my data is from the runtime package's "BenchmarkAllocation"; with GOGC=100 there are hundreds or thousands of GCs per second, so most of what it's benchmarking is the effect of STW delays on application throughput.)

For the stop-the-world events, I see that most of the wall-clock cost is from bringing the world to a stop, rather than slow bookkeeping in the STW's critical section.

I also see that forEachP, especially as used during GC phase transitions, causes pileups on sched.lock by 1/ holding it for a long time, while 2/ inducing demand for it on other Ms.

We should reduce the wall-clock time required for ragged barriers and to stop-the-world.

Some STWs see delay because after some Ps have begun to transition to "idle", a "running" P can still pick up and start running a G. The delay between the announcement of the STW and the start of that goroutine's run can be in the neighborhood of 50µs.

The path for a spinning M to steal a P's runnext slot includes a delay. The runnext slot enables cheap context switches between goroutines (which share a time slice) within a P, and unnecessary migration would defeat that optimization. Platforms with "fast" timers implement the delay as usleep(3). On Linux, the kernel pads that with some timer slack. In my testing, calls to usleep(3) on linux/amd64 usually take 50–60µs.

The spinning M verifies that the target P is running before committing to that sleep. The problem arises when a STW begins during the sleep. The spinning M wakes up, easily steals the runnext G (the victim P is now idle, honoring the STW request), and starts running it.

Meanwhile, the P which announced the STW has started its own sleep of 100µs (plus timer slack). When it wakes, it calls preemptall which eventually interrupts the stolen G.

The spinning M should verify conditions after its usleep(3) completes.

Even if it doesn't find work to steal, a spinning M's usleep(3) call can delay the STW. We should interrupt those sleeps as part of announcing the STW.

Here's an example from linux/amd64 of a spinning M (on P14) picking up new work after other Ps have already stopped:

Here's an example from linux/amd64 of a spinning M (on P19) taking more than 50µs to acknowledge the STW:

In forEachPInternal and stopTheWorldWithSema, we hold sched.lock while calling preemptall. The first problem with that is the length of the critical section. Each underlying call to signalM does a syscall (or two, since Linux does getpid and then tgkill), adding up to tens of microseconds while the caller holds sched.lock.

The second problem is that when a thread receives the preemption signal, the response we're asking it to have requires it to take sched.lock. When the mutex isn't available, and doesn't become available within a few microseconds, the thread goes to sleep. That results in more latency (fewer Ps able to run Gs) since the thread needs to be woken up and find its way back onto a CPU before it can finish the coordination work (parking, or running the safe-point function).

We should restructure those functions to schedule their calls to preemptall for a time when they're not holding sched.lock.

On Linux, we should call getpid once at startup and cache the result for signalM to use in its tgkill calls.

Here's a view of some of those pileups from the same linux/amd64 machine, with sched.lock instrumented by a low-overhead lock/unlock tracer. (I designed and conjured it; not ready to share.) The "Ms waiting" track shows the speed of the pileup and subsequent drain, while the "lock holder" track shows M17 in a critical section for 15–25µs at a time. Each major time division is 50µs, showing a bit over 200µs across the page.

Another problem with forEachPInternal is that it holds sched.lock while running the safe-point function on behalf of each idle P. CPU profiling shows that the cost of runtime.(*mcache).prepareForSweep (part of mark termination's safe-point function) for idle Ps is the next largest cause (after syscalls) of long sched.lock holds. I don't have crisp data on that at the moment.

It would be good to find a way to restructure forEachP so it doesn't have to hold sched.lock while running any safe-point functions.

CC @golang/runtime