runtime: TOCTOU race in scanConservative causes SIGSEGV via nil allocBits

[jonegenh]

runtime: TOCTOU race in scanConservative causes SIGSEGV via nil allocBits

Related: #39499, #47415

Go version

Confirmed: go1.17.5, go1.17.8 (linux/amd64).
Code-path still present in Go 1.26.4 (latest) and tip — unfixed.

What operating system and processor architecture are you using?

Linux amd64 (CentOS 7, kernel 4.19). Reproducible on multiple independent hosts.

What did you do?

We operate long-running Go services that use a log SDK with lz4 compression.
The SDK's PutRawLog function allocates a large stack variable (var hashTable [1<<16]int,
524KB) and runs a tight inline hash loop with no function calls (no GC safe points).

After running for hours to days under production load, the processes crash with SIGSEGV
in the GC conservative scan path:

runtime.scanConservative → runtime.(*mspan).isFree → SIGSEGV (allocBits=nil)

We observed this crash on multiple independent programs running on different machines,
confirming it is not hardware-specific or application-specific.

What did you expect to see?

No crash. The GC should safely handle conservative stack scanning even when spans
are concurrently being freed and reinitialized.

What did you see instead?

Crash variant 1: SIGSEGV (allocBits=nil dereference)

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x568 pc=0x41d114]

runtime stack:
runtime.throw({0x4be7b7, 0x43333e})
        runtime/panic.go:1198 +0x71
runtime.sigpanic()
        runtime/signal_unix.go:719 +0x396
runtime.(*mspan).isFree(...)
        runtime/mbitmap.go:226
runtime.scanConservative(0xc00207fdf0, 0x178, 0x0, 0x539888, 0x7f414e40cb90)
        runtime/mgcmark.go:1376 +0x134
runtime.scanframeworker(...)
        runtime/mgcmark.go:886 +0x158
runtime.scanstack(...)
        runtime/mgcmark.go:748
runtime.markroot(...)
        runtime/mgcmark.go:205
runtime.gcDrain(...)
        runtime/mgcmark.go:1013
runtime.gcBgMarkWorker.func2()
        runtime/mgc.go:1269

addr=0x568 = nil + objIndex/8, confirming span.allocBits was nil when isFree
dereferenced it via bitp().

Crash variant 2: zombie object (allocBit=0 + gcmarkBit=1)

runtime: marked free object in span 0x7f3fb5b80668, elemsize=16 freeindex=10
fatal error: found pointer to free object

A freed object was incorrectly marked as live during conservative scan, detected
during GC sweep as allocBit=0 (free) + gcmarkBit=1 (marked).

Root Cause Analysis

TOCTOU (Time-of-Check-to-Time-of-Use) race in scanConservative.

In runtime/mgcmark.go, the scanConservative function:

span := spanOfHeap(val)       // CHECK: span.state == mSpanInUse (time T1)
if span == nil {
    continue
}
// *** NO STATE RECHECK — TOCTOU gap ***
idx := span.objIndex(val)
if span.isFree(idx) {         // USE: dereferences span.allocBits (time T2)
    continue                   // if allocBits became nil → SIGSEGV
}

spanOfHeap (mheap.go) checks span.state.get() == mSpanInUse.
isFree (mbitmap.go) dereferences span.allocBits without re-verifying state.

Between these two reads (3-5 instructions, ~2-5ns), no lock is held.
The span can complete its entire lifecycle transition on another thread:

Thread A (GC scanner)          Thread B (allocator/sweeper)
─────────────────────────      ─────────────────────────────
spanOfHeap(val):
  span.state == mSpanInUse ✓
  returns span
                               freeSpanLocked(span):
     ═══ TOCTOU gap ═══         span.state = mSpanDead
                                 freeMSpanLocked(span)
                               allocSpan → tryAllocMSpan:
                                 gets same mspan struct
                                 s.init(base, npages):
                                   s.allocBits = nil  ← !!!
                                   s.state = mSpanDead
span.isFree(idx):
  allocBits.bitp(idx)
  → nil dereference → SIGSEGV

Why the existing defense is insufficient

The allocSpan code in mheap.go contains this comment:

// Now that the span is filled in, set its state. This
// is a publication barrier for the other fields in
// the span. While valid pointers into this span
// should never be visible until the span is returned,
// if the garbage collector finds an invalid pointer,
// access to the span may race with initialization of
// the span. We resolve this race by atomically
// setting the state after the span is fully
// initialized, and atomically checking the state in
// any situation where a pointer is suspect.
s.state.set(mSpanInUse)

This defense protects the allocation path (Init→InUse): if the scanner reads
state=mSpanInUse, it is guaranteed that allocBits has already been written
(x86 TSO + publication order).

But the TOCTOU bug is on the deallocation path (InUse→Dead→reinit). The scanner
reads state=mSpanInUse at time T1 (old value, before the span was freed). Then at
time T2, it reads allocBits=nil (new value, after reinitialization). Both reads
observe correctly-ordered stores — but from different lifecycle phases of the
same span struct. This is a classic TOCTOU, not a memory ordering issue.

Why x86 TSO does not help

On x86, stores are visible in program order. So:

• If state=InUse is observed at T1, allocBits set before that state is also visible — but only the old allocBits, not the new nil from a reinit.
• After the span transitions through Dead→reinit, allocBits=nil is stored before the new state=InUse. The scanner reading old state=InUse and new allocBits=nil is not a TSO violation; it's two independent loads at different times.

Trigger conditions (all required simultaneously)

1. Large stack frame with residual heap addresses: A [1<<16]int (524KB) stack-allocated
   hash table. Due to stack reuse, ~37% of slots contain stale values that may happen
   to be valid heap addresses.

2. Tight loop without safe points: The hash computation loop has no function calls,
   preventing Go from inserting cooperative GC safe points.

3. Async preemption (SIGURG) → conservative scan: The runtime sends SIGURG to preempt
   the goroutine mid-loop. scanframeworker detects the frame below asyncPreempt
   has no precise pointer map → falls back to conservative scanning.

4. Conservative scan treats all frame values as potential pointers: All 65536 slots
   are examined; ~0.6% pass spanOfHeap's validity checks.

5. Span lifecycle churn: Under GOGC=1 or heavy allocation, spans frequently cycle
   through InUse→Dead→reinit, creating brief allocBits=nil windows.

Production evidence

We have observed this crash on 5+ independent hosts running different Go services
(log SDK with lz4, gRPC services with cgo) over 6+ months, with identical stack traces.
All hosts run linux/amd64, Go 1.16–1.17, under production traffic. The crash pattern
matches the earlier report #47415 ("runtime: marked free object in span", Go 1.16.6,
cgo + gRPC, closed without resolution due to inability to reproduce), confirming this
is a long-standing production issue that has affected multiple users independently.

Reproduction with runtime patch

The natural TOCTOU window is 2-5ns and crashes require hours to days in production.
The runtime patch below does not introduce the bug — it merely widens the natural
allocBits=nil window (which exists on every mspan.init() call) from ~2-5ns to 50μs,
making the race deterministically hittable for reviewer convenience.

To reproduce deterministically, we patch runtime/mheap.go (allocSpan, after
s.state.set(mSpanInUse)) to widen the allocBits=nil window:

Patch (mheap.go, allocSpan function, after s.state.set(mSpanInUse)):

 		s.state.set(mSpanInUse)
+		// TOCTOU reproduction patch: create InUse + allocBits=nil window.
+		// Simulates the natural race where a concurrent thread frees and
+		// reinitializes the span between spanOfHeap and isFree.
+		tmp := s.allocBits
+		s.allocBits = nil
+		usleep(50)
+		s.allocBits = tmp
 	}

Result: Crash within seconds (vs. hours/days without patch):

$ GOGC=1 GOTRACEBACK=crash ./gc_repro_bin
Go go1.17.5  GOMAXPROCS=4  GOGC=1  workers=32
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x568 pc=0x41d114]
runtime.(*mspan).isFree(...)
runtime.scanConservative(...)
runtime.scanframeworker(...)  ← conservative=true
runtime.scanstack(...)
runtime.gcBgMarkWorker(...)

The patch does not change the race mechanism — it merely makes the natural
allocBits=nil window (which exists during every span reinitialization) wide
enough to be reliably hit by the concurrent scanner.

Reproduction program: main.go (click to expand)

package main

// Reproduce scanConservative TOCTOU crash.
// Requires: Go 1.14+ (async preemption), linux/amd64, CGO_ENABLED=1
//
// Without runtime patch: crash in hours to days (use CPU pinning + SIGURG flood)
// With runtime patch: crash in seconds

/*
#include <unistd.h>
*/
import "C"

import (
	"fmt"
	"os"
	"os/signal"
	"runtime"
	"runtime/debug"
	"sync/atomic"
	"syscall"
	"time"
	"unsafe"
)

var (
	iter   uint64
	poison [1 << 16]uintptr
)

// seedPoison fills the poison array with valid heap addresses.
// After the objects are freed (GC'd), these become stale pointers that
// conservative scan will treat as potential heap references.
func seedPoison() {
	objs := make([][]byte, len(poison))
	for i := range objs {
		objs[i] = make([]byte, 48)
		poison[i] = uintptr(unsafe.Pointer(&objs[i][0]))
	}
	objs = nil
	runtime.GC()
	runtime.GC()
}

// refreshPoison continuously allocates and records heap addresses,
// keeping poison[] populated with addresses that span various size classes.
func refreshPoison() {
	for {
		for i := range poison {
			obj := make([]byte, 48)
			poison[i] = uintptr(unsafe.Pointer(&obj[0]))
		}
		runtime.Gosched()
	}
}

//go:noinline
func putRawLog(body []byte) {
	// 524KB stack-allocated hash table — simulates lz4 compression in log SDKs.
	var hashTable [1 << 16]int

	// Fill with stale heap addresses (simulates stack reuse residual data).
	dst := (*[1 << 16]uintptr)(unsafe.Pointer(&hashTable[0]))[:]
	copy(dst, poison[:])

	// Tight hash loop with NO function calls = NO safe points.
	// SIGURG async preemption catches the goroutine here,
	// triggering conservative scan of the 524KB hashTable frame.
	for i := 0; i+4 <= len(body); i++ {
		h := uint(body[i]) | uint(body[i+1])<<8 | uint(body[i+2])<<16 | uint(body[i+3])<<24
		idx := (h * 2654435761) >> 16 & 0xFFFF
		hashTable[idx] = i
	}

	// Keep frame alive to extend conservative scan window.
	sum := 0
	for i := 0; i < 1<<16; i++ {
		sum += hashTable[i]
	}
	if sum == -0xdead {
		println("unreachable")
	}
	runtime.KeepAlive(&hashTable)
}

func main() {
	debug.SetGCPercent(1)
	runtime.GOMAXPROCS(runtime.NumCPU())

	body := make([]byte, 64*1024)
	for i := range body {
		body[i] = byte(i)
	}

	seedPoison()

	// Refresh poison addresses continuously.
	for i := 0; i < 4; i++ {
		go refreshPoison()
	}

	// CGO calls increase OS thread count and scheduling complexity.
	for i := 0; i < 4; i++ {
		go func() {
			for {
				C.getpid()
				runtime.Gosched()
			}
		}()
	}

	// Span churn: allocate/free objects across many size classes to force
	// span lifecycle transitions (InUse → Dead → reinit).
	spanSizes := []int{48, 64, 80, 96, 112, 128, 144, 160, 176, 192,
		208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480}
	for i := 0; i < runtime.NumCPU()*4; i++ {
		go func(id int) {
			for {
				sz := spanSizes[(id+int(atomic.LoadUint64(&iter)))%len(spanSizes)]
				batch := make([][]byte, 256)
				for j := range batch {
					batch[j] = make([]byte, sz)
				}
				runtime.Gosched()
			}
		}(i)
	}

	// GC pressure: large allocations force frequent GC at GOGC=1.
	for i := 0; i < 4; i++ {
		go func() {
			for {
				_ = make([]byte, 1<<20)
			}
		}()
	}

	// putRawLog workers.
	w := runtime.NumCPU() * 8
	if w < 32 {
		w = 32
	}

	fmt.Printf("Go %s  GOMAXPROCS=%d  GOGC=1  workers=%d  cgo=enabled\n",
		runtime.Version(), runtime.GOMAXPROCS(0), w)
	fmt.Printf("Target: scanConservative TOCTOU race → allocBits=nil → SIGSEGV\n\n")

	for i := 0; i < w; i++ {
		go func() {
			for {
				putRawLog(body)
				atomic.AddUint64(&iter, 1)
			}
		}()
	}

	sig := make(chan os.Signal, 1)
	signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
	tk := time.NewTicker(5 * time.Second)
	start := time.Now()
	for {
		select {
		case <-tk.C:
			n := atomic.LoadUint64(&iter)
			var m runtime.MemStats
			runtime.ReadMemStats(&m)
			fmt.Printf("t=%ds  iter=%d  gc=%d  heap=%dMB  sys=%dMB\n",
				int(time.Since(start).Seconds()), n, m.NumGC, m.HeapAlloc>>20, m.Sys>>20)
		case <-sig:
			return
		}
	}
}

Running without patch (hours to days):

# Build
CGO_ENABLED=1 go build -o gc_repro_bin .

# CPU contention + SIGURG flood (recommended for faster reproduction)
ulimit -c unlimited
export GOGC=1 GOTRACEBACK=crash
taskset -c 0,1 ./gc_repro_bin &
PID=$!
taskset -c 0,1 bash -c 'while true; do :; done' &   # CPU stress on same cores
while kill -URG $PID 2>/dev/null; do :; done &       # SIGURG flood

# Wait for crash (minutes to hours with pinning, hours to days without)

Suggested Fix

Recheck span state before accessing allocBits in scanConservative:

// mgcmark.go, in scanConservative:
span := spanOfHeap(val)
if span == nil {
    continue
}
idx := span.objIndex(val)
// Fix: recheck state to close TOCTOU window.
// If the span was freed between spanOfHeap and here, skip it.
if span.state.get() != mSpanInUse {
    continue
}
if span.isFree(idx) {
    continue
}

This narrows the window significantly (though doesn't eliminate it entirely due to
the remaining gap between the recheck and isFree). A more robust fix would be to
make isFree check for nil allocBits:

func (s *mspan) isFree(index uintptr) bool {
    if index < s.freeindex {
        return false
    }
    bits := s.allocBits
    if bits == nil {
        return true // span is being reinitialized, treat as free
    }
    bytep, mask := bits.bitp(index)
    return *bytep&mask == 0
}

Affected versions

Version range	Impact
Go 1.14 – 1.18	Full exposure: async preemption + no loop safe points
Go 1.19 – 1.21	Reduced: loop safe points added (most tight loops get preemption points), but CGO frames, assembly, and //go:nosplit functions still trigger conservative scan
Go 1.22+	Further reduced: improved register maps, but scanConservative code path still exists and TOCTOU gap remains
Go 1.26.4 (latest)	Still vulnerable: scanConservative → spanOfHeap → isFreeOrNewlyAllocated has identical TOCTOU gap

Code references (Go tip, as of June 2026)

• runtime/mgcmark.go: scanConservative — TOCTOU between spanOfHeap and isFreeOrNewlyAllocated
• runtime/mheap.go: spanOfHeap — state check only, no allocBits validation
• runtime/mheap.go: allocSpan — comment acknowledging the race (line ~1255)
• runtime/mheap.go: mspan.init — sets allocBits=nil, state=mSpanDead
• runtime/mheap.go: freeSpanLocked — sets state=mSpanDead last (allocBits still valid during free, but nil after reinit)
• runtime/mbitmap.go: isFreeOrNewlyAllocated — dereferences allocBits without nil check

Related issues

Issue	Relation
#39499	Conservative stack scanning introduced with async preemption (Go 1.14). This is the mechanism that exposes the TOCTOU gap — conservative scan treats all frame values as potential pointers and calls spanOfHeap + isFree on each.
#47415	"runtime: marked free object in span" — Go 1.16.6, cgo + gRPC service, identical "Crash variant 2" symptom (allocBit=0 + gcmarkBit=1). Closed without resolution (unable to reproduce). We believe this was the same TOCTOU race manifesting in production, but at the time no root cause analysis was performed. Our deterministic reproduction and root cause analysis now explain why that crash occurred.

[gabyhelp]

Related Issues

• runtime: SIGSEGV on nil pointer in mheap.freeManual #73628 (closed)
• runtime: scanobject can use incorrect span metadata #11267 (closed)
• runtime: malloc crashes with 4TB arena #20882 (closed)
• runtime: heap `mspan` limit is set too late, causing data race between span allocation and conservative scanning #74288 (closed)
• runtime: segfault during conservative scan of asynchronously preempted goroutine #39499
• runtime: SIGSEGV in runtime.(*fixalloc).alloc #47302 (closed)
• runtime: `go runtime.GC()` can cause segfault with `-race` builds #78059 (closed)
• x/tools/gopls: crash in bgsweep #73262 (closed)
• runtime: panic marked free object in span[1.17] #54247 (closed)
• runtime: solaris/amd64 crash in garbage collector #7554 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[thepudds]

Hi @jonegenh, thanks for the report.

I have not fully digested the reproducer or everything you wrote here, but some quick suggestions:

1. From a quick look, I'll note the reproducer has a clear data race on poison[i] = uintptr(unsafe.Pointer(&obj[0])). Maybe that is somehow unimportant to what is being demonstrated, but it would be best if the reproducer did not have a data race and if you could confirm that by running for a period of time under -race.

2. My understanding is that this was observed in older releases circa Go 1.17 and that you are saying code inspection/analysis suggests it might still be present in the latest releases. However, it would be good if you try could reproducing the problem with Go 1.25 or Go 1.26 to confirm that empirically, if you haven't already. (There have been many improvements in a variety of areas since Go 1.17. Normally, a problem that only affects older releases would be closed per the policy on supported Go releases).

3. Finally, it would be helpful if you could also see whether GODEBUG=asyncpreemptoff=1 causes the problem to stop happening with your reproducer. The results of that likely would be a useful hint.

[JunyangShao]

@golang/runtime

[randall77]

That does look like a problem. Maybe before conservative scanning in a span, we need to temporarily acquire sweep rights to it, so it is not swept out from under us. That might introduce its own problems though, not sure.
Conservative scanning is a royal pain.

[mknyszek]

scanConservative cannot run concurrently with sweeping. They're executing in different GC phases.

[mknyszek]

FWIW, I believe you have encountered a real issue. We have fixed bugs with conservative scanning. As Keith says, it is a pain. But you're running Go 1.17, and we had several fixes around exactly this kind of issue in more recent releases. I suspect the issue you're seeing is that scanConservative is observing a partially-initialized span (an issue we explicitly fixed, see https://go.dev/cl/672340), rather than concurrent sweeping and scanning (that would be a much larger issue). This explains why allocBits is nil. I suspect you will not be able to reproduce with a newer release.

[mknyszek]

Oops, I missed the reproducer because it was hidden in a <details>. Apologies. There may well be some other issue here. But I'll just point out again that the root cause doesn't make sense. The sweeper is designed such that we only allow the mark phase to start if all sweepers have left sweep (sweepLocker ensures this). The mark phase similarly also only completes if all workers have gone to sleep (the nwait check in gcMarkDone).

And on second thought, https://go.dev/cl/672340 only helps with partially-initialized objects. Observing a partially-initialized span should not be possible in today's code because spanOfHeap checks for mSpanInUse. I'm actually not sure how it's possible to observe allocBits as nil -- that's still a puzzle.

[mknyszek]

Sorry, last thought: I think the runtime patch here might also be wrong and introduces a bug. It's violating an invariant in the allocator, specifically that the span is fully initialized BEFORE we set mSpanInUse. I'm not surprised that it forces a crash: spanOfHeap checks for mSpanInUse to make sure the span is safe to access (such as allocBits being set), but that patch is clearly working around that.