#!stacks
"sigpanic" && "(*spanSet).pop:+48,+0x118" && "runtime.bgsweep"
// Given that the spine length is correct, we know we will never
// see a nil block here, since the length is always updated after
// the block is set.
block := blockp.Load()
s := block.spans[bottom].Load() // signal!
for s == nil {
// We raced with the span actually being set, but given that we
// know a block for this span exists, the race window here is
// extremely small. Try again.
s = block.spans[bottom].Load()
}golang.org/x/tools/gopls@v0.21.1 go1.25.6 darwin/arm64 gopls.client=vscode-insiders (1)
I'm not familiar with this part of the runtime, but here's Gemini's guess as to the cause:
Based on the telemetry stack trace and the hardware-fault-investigator workflow, here is the step-by-step investigation:
1. Extracted Context
- Go Version: go1.25.6
- Target OS/Arch: darwin/arm64
- Application: golang.org/x/tools/gopls@v0.21.1
- Crashing Function: runtime.(*spanSet).pop
- Crashing Offset: +0x118
2. Disassembly & Identification
I built gopls locally matching the exact toolchain and architecture to extract the assembly of runtime.(*spanSet).pop.
The function starts at 0x10003b8a0. The offset +0x118 corresponds to instruction 0x10003b9b8. Here is the relevant disassembly block:
1 // blockp := b.spine.Load().lookup(uintptr(top))
2 0x10003b990 d3497c82 UBFX $9, R4, $23, R2 // R2 = top (head >> 9)
3 // ... calculates spine offset ...
4 0x10003b9a4 8b020c82 ADD R2<<3, R4, R2 // R2 = &spine.spine[top] (blockp)
5
6 // block := blockp.Load()
7 0x10003b9b4 c8dffc41 LDAR (R2), R1 // R1 = blockp.Load()
8
9 // s := block.spans[bottom].Load()
10 0x10003b9b8 3980003b MOVB (R1), R27 // CRASH HERE (+0x118)
11 0x10003b9bc 91080024 ADD $512, R1, R4
12 0x10003b9c0 8b030c83 ADD R3<<3, R4, R3 // Computes &block.spans[bottom]
3. Trace and Analysis
1. The Crashing Instruction: The sigpanic occurred at MOVB (R1), R27. This is a compiler-emitted implicit nil check before computing the offset for
block.spans[bottom]. It attempts to dereference R1, meaning R1 holds an invalid memory address (specifically 0x0 or nil).
2. Tracing R1: R1 is loaded directly from R2 (blockp.Load()) via LDAR (R2), R1 right before the crash.
3. Validating Preconditions: For R1 (the block pointer) to be nil, blockp.Load() must have returned nil.
4. Identifying "Impossible" States (Software Race Condition)
Can blockp.Load() return nil in pure software?
Yes, due to a known time-of-check to time-of-use (TOCTOU) concurrency flaw in the GC's spanSet design:
1. pop() claims a span by incrementing the lock-free queue's head via CompareAndSwap. At this exact moment, head may become equal to tail (queue is empty).
2. The goroutine executing pop() can be preempted by the OS scheduler immediately after the CAS, but before it executes block := blockp.Load().
3. Meanwhile, the background GC sweeper relies on spanSet.empty() (which simply checks head >= tail) to determine if sweeping is complete. Since the stalled
pop() already CASed head, the GC assumes sweeping is 100% complete and signals isSweepDone().
4. The Go runtime transitions to a new GC cycle. As part of GC initialization, it calls spanSet.reset() on the now "empty" spanSet.
5. spanSet.reset() iterates through the spine array, explicitly executing blockp.StoreNoWB(nil) and freeing the underlying blocks.
6. The original pop() goroutine finally wakes up from its stall. It proceeds to execute LDAR (R2), R1 (block := blockp.Load()), reading the nil freshly
deposited by reset(). It then executes the MOVB instruction and crashes.
5. Determine the Cause
Verdict: Software Bug
The crash is not caused by a hardware failure. It is a pure software race condition in the Go runtime's garbage collector. The lock-free design allows pop()
to signal that an element has been processed before it has actually read the pointer, permitting a concurrent GC rollover (reset()) to destroy the data
structures underneath a stalled goroutine.
Issue created by stacks.
This stack
1Tnp4Qwas reported by telemetry:crash/crashruntime.throw:+9,+0x33runtime.sigpanic:+3,+0x34fruntime.(*spanSet).pop:+48,+0x118runtime.(*mheap).nextSpanForSweep:+9,+0xc3runtime.sweepone:+19,+0xd7runtime.bgsweep:+28,+0x12fruntime.gcenable.gowrap1:+0,+0x27runtime.goexit:+0,+0x3Use this command to reproduce the executable:
(HOME=$(mktemp -d); GOOS=darwin GOARCH=arm64 GOTOOLCHAIN=go1.25.6 go install golang.org/x/tools/gopls@v0.21.1 && find $HOME/go/bin -type f)To disassemble:
go tool objdump exe | lessI'm not familiar with this part of the runtime, but here's Gemini's guess as to the cause: