Repository navigation
runtime: GC stack shrinking is O(n²) in the number of goroutines blocked on one sync.Mutex since Go 1.26 #81957
Description
Go version
1.26.8 and 1.27.1 on both linux/arm64 & darwin/arm64
Output of go env in your module/workspace:
GOARCH='arm64'
GOOS='linux'
GOVERSION='go1.26.8'
GOEXPERIMENT=''
GODEBUG=''
GOARM64='v8.0'
CGO_ENABLED='1'What did you do?
We run an HTTP reverse proxy built with Go 1.26. Under a traffic spike, thousands of goroutines blocked in sync.Mutex.Lock inside net/http.(*Transport), and the process spent half its CPU in GC. In a 30s CPU profile from the Linux production host (113.8s of samples):
runtime.adjustsudogswas 48.9s, 43% of all CPU, throughmarkroot→scanstack→shrinkstack→copystack→adjustsudogs→adjustpointer.- Without it, GC would have been about 7s. The walk was about 87% of all GC work.
- 25.9s of the 49.3s in
shrinkstackran insidegcAssistAlloc, so request goroutines paid for it directly.
Reproducer: n goroutines grow their stacks, return, and block on one sync.Mutex. Automatic GC is off, and the program calls 4 times runtime.GC(). It measures process CPU time (getrusage).
Code: https://go.dev/play/p/mflVhUvT3uO
GOTOOLCHAIN=go1.25.14 go run . -n 40000
GOTOOLCHAIN=go1.26.8 go run . -n 40000
What did you see happen?
All numbers are process CPU time for the 4 timed GCs (sum), on linux/arm64 unless noted.
| goroutines blocked on 1 mutex | go1.25.14 | go1.26.8 |
|---|---|---|
| 5,000 | 111 ms | 233 ms |
| 10,000 | 203 ms | 698 ms |
| 20,000 | 391 ms | 2,412 ms |
| 40,000 | 751 ms | 9,755 ms |
| 80,000 | 1,541 ms | 45,107 ms |
When the waiter count doubles, go1.25 grows about 2×. go1.26 grows 3.0×, 3.5×, 4.0×, then 4.6×.
Controls at 40,000 waiters (3 alternating runs where a range is given):
| build | total GC CPU |
|---|---|
| go1.25.14 | 770–782 ms |
| go1.26.8 | 9,555–9,885 ms |
go1.26.8, GODEBUG=gcshrinkstackoff=1 |
625 ms |
| go1.27.1 | 9,670–10,877 ms |
go1.27.1, GODEBUG=gcshrinkstackoff=1 |
595 ms |
darwin/arm64 shows the same pattern at 40,000 waiters: go1.25.14 0.79–0.96s, go1.26.8 7.1–7.2s.
What did you expect to see?
GC stack-shrink cost that grows linearly with the number of blocked goroutines, like in Go 1.25, not growing exponentially.
It looks like this is caused by the commit 8c68a1c1ab (CL 688335, goroutine leak detection, #74609) added this line to semaRoot.queue (sema.go:309 at go1.26.8):
// Storing this pointer so that we can trace the semaphore address
// from the blocked goroutine when checking for goroutine leaks.
s.g.waiting = sRemoving that one line brings go1.26.8 back to go1.25 levels.
| build | total GC CPU |
|---|---|
| go1.25.14 | 770–782 ms |
| go1.26.8 | 9,555–9,885 ms |
go1.26.8 with only the s.g.waiting = s line in semaRoot.queue removed |
786–796 ms |
Activity
gabyhelp commentedon Oct 2, 2026
Related Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
thepudds commentedon Oct 2, 2026
CC @VladSaiocUber (for goroutine leak detection) @golang/runtime
gopherbot commentedon Oct 3, 2026
Change https://go.dev/cl/844405 mentions this issue:
runtime: avoid scanning semaphore queues during stack copyingprattmic commentedon Oct 6, 2026
Thanks for the report, I can reproduce this, it definitely needs a fix.
1 remaining item
mknyszek commentedon Oct 6, 2026
Oof, nice catch.
However, I'm not confident in the correctness and maintainability of the fix in https://go.dev/cl/844405: that seems very subtle at face value. I think we might want to just look for another way to make goroutine leak detection able to discover the semaphore address than via
gp.waiting.prattmic commentedon Oct 6, 2026
I tend to agree, though I don't yet understand why we don't see the same quadratic behavior for goroutines blocked on channels. Those look almost perfectly linear in my empirical testing.
prattmic commentedon Oct 6, 2026
Ah, I see.
For channels,
sudog.waitlistis the list of channels that the goroutine is waiting on. Eachsudogis for the same goroutine, but different channel.For semaphores,
sudog.waitlistis the list of goroutines that are waiting on the semaphore. Eachsudogis for the same semaphore, but different goroutines.So if every goroutine is waiting on the same semaphore, each
gp.waitingwe scan discovers every goroutine.It also seems like this is unsafe? If a semaphore unblocks while we are in adjustsudog, semaRoot.dequeue will concurrently modify the sudog waitlist while we are walking over it. This isn't possible in the channel case because the goroutine can't unblock from a channel while it is in copystack. I don't think this race matters though, since the semaphore address will never be on the stack.
prattmic commentedon Oct 7, 2026
Semaphores can't be on the stack, so they don't need adjustment at all (e.g.,
runtime_SemacquireMutexleaksMutex.semato the heap).I see two obvious directions here:
One,
adjustsudogsdetects thesudogtype and skips everything except channels (basically what https://go.dev/cl/844405 does, but could be cleaner).Alternatively,
gp.waitingonly stores channelsudogs and semaphores are stored elsewhere (gp.waitingChan/gp.waitingSema). This latter approach requires more space in the G, but I do think it is more understandable and thus maintainable, so I lean towards that.In fact, I think
sudogis a mess. It is used for channels, semaphores, and sync.Cond, and all three uses are almost completely disjoint, using different subsets of fields with different semantics. This is how we ended up withwaitlisthaving completely different semantics between channels and semaphores. I am strongly considering splittingsudoginto three distinct types for each of the different uses.gopherbot commentedon Oct 8, 2026
Change https://go.dev/cl/847405 mentions this issue:
runtime: throw if unswept spans remain in finishsweep_mgopherbot commentedon Oct 8, 2026
Change https://go.dev/cl/847406 mentions this issue:
runtime: only adjust sudogs for channelsruntime: throw if unswept spans remain in finishsweep_m
prattmic commentedon Oct 13, 2026
@gopherbot Please backport https://go.dev/cl/847406 to Go 1.26 and 1.27. This is a performance regression making stack shrinking O(n^2) on the number of goroutines blocked on the same sync.Mutex. It can make the GC use extremely high CPU.
This regression was introduced in https://go.dev/cl/688335, which introduced
GOEXPERIMENT=goroutineleakprofile. This GOEXPERIMENT was disabled by default in 1.26, but the regression is present whether the experiment is enabled or disabled, so 1.26 is affected as well.gopherbot commentedon Oct 13, 2026
Backport issue(s) opened: #82189 (for 1.26), #82190 (for 1.27).
Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.
gopherbot commentedon Oct 13, 2026
Change https://go.dev/cl/849145 mentions this issue:
Revert "runtime: throw if unswept spans remain in finishsweep_m"gopherbot commentedon Oct 13, 2026
Change https://go.dev/cl/849146 mentions this issue:
runtime: update reason for unswept spans remaining in finishsweep_mRevert "runtime: throw if unswept spans remain in finishsweep_m"
runtime: update reason for unswept spans remaining in finishsweep_m