runtime: selecting mark workers during start-the-world (CL 721002) causes RSS regression in gopher-lua on large machines

[prattmic]

https://perf.golang.org/dashboard/?benchmark=GopherLuaKNucleotide-72&unit=peak-RSS-bytes&repository=go&branch=master&platform=linux%2Farm64&days=60&end=2025-12-19T23%3A15 shows a peak-RSS-bytes regression for gopher-lua at high CPU counts on 2025-11-20.

The exact commit is unclear from the dashboard, but bisection on a 384 core machine blames https://go.dev/cl/721002 (runtime: select GC mark workers during start-the-world).

The exact cause of the regression is still unknown.

cc @golang/runtime @mknyszek

[prattmic]

CPU profile before:

CPU profile after:

Nothing to see here, move along...

[prattmic]

A runtime trace confirms my initial suspicion that the GC is simply too fast.

The before case only has 32 procs and 34 threads total. The after case has 121 procs and 130 threads.

The GC targets 25% utilization. With GOMAXPROCS=384, that is 96 procs. The after case is behaving about as expected.

The before case has far fewer procs than expected. We can nicely see what happens here:

Before https://go.dev/cl/721002, the flow starting the GC looked like:

1. Start the world sees that there is 1 runnable goroutine, so it wakes one P.
2. This P realizes it needs to be a dedicated GC worker, so it starts doing that and wakes another P in case there is more work (there is).
3. The next P also needs to be a dedicated GC worker, so it starts another P.
4. This process continues incrementally for ~1.5ms, at which point the existing GC workers manage to complete the entire concurrent mark phase before all of the workers manage to start.
5. Since the mark phase is over, the next P to start does not need to run a GC worker and can continue the application goroutine.

After https://go.dev/cl/721002, start the world starts all of the Ps needed for dedicated workers at once, and all 96 do manage to start before the mark phase completes. (The extra CPU time in the profile above is a result of poor scalability in the GC on this many cores, which is tracked by other bugs).

The additional RSS is presumably from the additional memory required for all of the extra Ps and Ms. The absolute RSS is not very high (~65MB before and ~90MB after).

Personally I don't believe there is anything we need to do here imminently. Real applications running at such high scale presumably have much larger heaps and would thus have been running all GC workers before or after this change.

There is perhaps a long-term opportunity to consider pacing changes for small heaps where the overhead of starting 25% of Ps likely outweighs the short wall-time we expect marking to take.

[prattmic]

One more observation:

In the before case, "stop-the-world (GC sweep termination)" is ~100us. In the after case, it is ~800us.

We can also still see a "wavefront" of GC workers starting in the after case, despite starting them "all at once":

As far as I can tell, this is just startTheWorld literally looping over all runnable Ps and waking their M, which has to happen in sequence. The absolute difference between sequential P wakeup times is only ~4us.

This is also technically after the world is started. So even though it appears that the STW time is ~800us, most of that time is actually spent waking Ps which are able to run goroutines as they wake up.

runtime/metrics already has /sched/pauses/stopping/gc:seconds and /sched/pauses/total/gc:seconds to distinguish between time "stopping the world" when some Ps are still running, and time spent fully stopped. It seems like we may also want /sched/pauses/starting/gc:seconds to show time spent partially restarted.