runtime: long stop-the-world due to scavenging when exceeding GOMEMLIMIT

[nsrip-dd]

We have a program, built with Go 1.24.2, where we sometimes see very long stop-the-world pauses. The program has GOGC=off and a GOMEMLIMIT of ~12GiB. We see the pauses when the program has exceeded its memory limit.

For example, this execution trace shows a 400ms stop-the-world pause. The goroutine in proc 1 is trying to stop the world for sweep termination, and a CPU profile sample shows that it is trying to preempt goroutines. Meanwhile, the goroutine on proc 4 is running for 400ms, and all the CPU profile samples during that time show it calling runtime.sysUnused during runtime.(*pageAlloc).scavenge, where it can't be preempted. The third screenshot shows that GC is happening almost constantly during the trace.

In our case, we can increase our memory limit if needed. I'm mainly filing this issue to document this worst-case behavior, in case there are things we could do to improve it.

cc @mknyszek

[mknyszek]

Much like how we back out after sweeping 100 spans in the allocator, we should bound the amount of time we spend here. We have two options:

1. If there's a pending preemption request, we 'fail' the malloc, unwind the stack, and force it to retry, or,
2. Proceed with the malloc anyway.

(2) is a bit easier to implement and doesn't have to affect the malloc fast path at all, since we don't need to add the ability to propagate up a failure, but has the downside that we may end up exceeding the memory limit. In this particular case, the program is already in a difficult place, and something has to give. If we can come up with some principled policy for why we back out when we do, (2) seems fine to me. (The sweeper sets the number at 100 spans because, in the worst case, we use 1% more memory, which is an arbitrary but reasonable number.)