Go version
go version go1.25.7 darwin/arm64
Output of go env in your module/workspace:
GOOS=darwin
GOARCH=arm64
CGO_ENABLED=1
macOS 15.7 (Apple M-series). The defect is present at tip by code inspection (runtime/race.go, isvalidaddr; see analysis below).
What did you do?
Ran the program below with -race. It does three things:
mmaps 1 MiB of anonymous memory with an address hint inside the race-mode heap window [0x00c000000000, 0x00e000000000). The hint only makes the placement deterministic — in real life the kernel does this unprompted once lower VA is crowded (see "How this happens in the wild" below; anonymous vs file-backed makes no difference to TSAN).- Grows the heap past that mapping, so
racearenaend extends beyond it.
- Performs one instrumented read of the mapping.
package main
/*
#include <stdint.h>
#include <sys/mman.h>
// map_at maps anonymous memory at the given hint. Not MAP_FIXED: the kernel
// honors the hint only if the range is free, else picks another address —
// same as any regular mmap(NULL) placement decision.
static uintptr_t map_at(uintptr_t hint, size_t len) {
return (uintptr_t)mmap((void *)hint, len, PROT_READ | PROT_WRITE,
MAP_ANON | MAP_PRIVATE, -1, 0);
}
*/
import "C"
import (
"fmt"
"runtime"
"unsafe"
)
const (
windowBeg = 0x00c0_0000_0000 // race-mode heap window, runtime/malloc.go
windowEnd = 0x00e0_0000_0000
mapLen = 1 << 20
)
func main() {
// 1. Place a non-runtime mapping inside the race heap window, above the
// current (small) heap.
const hint = windowBeg + 1<<28 // 256MiB above window base
p := uintptr(C.map_at(C.uintptr_t(hint), mapLen))
if p == ^uintptr(0) || p < windowBeg || p+mapLen > windowEnd {
fmt.Printf("could not place mapping inside the window (got %#x)\n", p)
return
}
fmt.Printf("squatter mapped at %#x (inside the race heap window)\n", p)
// 2. Grow the heap past the squatter so racearenaend extends beyond it.
// The runtime maps TSAN shadow per heap arena only, so the squatter's
// shadow range stays a hole.
var ballast [][]byte
for i := 0; i < 64; i++ {
b := make([]byte, 64<<20)
b[0] = 1
ballast = append(ballast, b)
top := uintptr(unsafe.Pointer(&b[len(b)-1]))
if top > p+mapLen {
fmt.Printf("heap grew past the squatter (alloc ends at %#x, %d MiB live)\n",
top, (i+1)*64)
break
}
}
runtime.KeepAlive(ballast)
// 3. One instrumented read of the squatter. isvalidaddr (runtime/race.go)
// passes it to TSAN — it is inside [racearenastart, racearenaend) — but
// its shadow was never mapped, so __tsan_read faults.
sum := byte(0)
for off := uintptr(0); off < mapLen; off += 4096 {
sum += *(*byte)(unsafe.Pointer(p + off))
}
fmt.Println("no crash; sum =", sum)
}Without -race it prints no crash; sum = 0. With -race it dies 100% deterministically.
What did you see happen?
$ go run -race .
squatter mapped at 0xc010000000 (inside the race heap window)
heap grew past the squatter (alloc ends at 0xc103ffffff, 256 MiB live)
runtime: newstack sp=0x16fa32d20 stack=[0xc0001b8000, 0xc0001ba000]
morebuf={pc:0x10043ed24 sp:0x16fa32d20 lr:0x0}
sched={pc:0x100438c2c sp:0x16fa32d20 lr:0x10043ed24 ctxt:0x0}
runtime: gp=0xc0000021c0, goid=1, gp->status=0x2
runtime: split stack overflow: 0x16fa32d20 < 0xc0001b8000
fatal error: runtime: split stack overflow
runtime stack:
runtime.throw({0x1004cdb73?, 0x1003f6ee0?})
.../src/runtime/panic.go:1094 +0x34 fp=0x16fa32c00 sp=0x16fa32bd0 pc=0x100437354
runtime.newstack()
.../src/runtime/stack.go:1100 +0x590 fp=0x16fa32d30 sp=0x16fa32c00 pc=0x10041fef0
runtime.morestack()
.../src/runtime/asm_arm64.s:392 +0x70 fp=0x16fa32d30 sp=0x16fa32d30 pc=0x10043bb70
goroutine 1 gp=0xc0000021c0 m=0 mp=0x1005c12a0 [running]:
runtime.sigpanic()
.../src/runtime/signal_unix.go:906 +0x35c fp=0x16fa32d20 sp=0x16fa32d20 pc=0x100438c2c
racecall()
.../src/runtime/race_arm64.s:484 +0x34 fp=0x16fa32d40 sp=0x16fa32d30 pc=0x10043ed24
...
Catching the original (non-reduced) fault under lldb shows what actually happens before the runtime mangles it: EXC_BAD_ACCESS inside __tsan_read, at the shadow address (app*2 + 0x200000000000) of a file mapping sitting between Go heap arenas — that shadow range is unmapped. The SEGV lands while racecall is running on the system stack, so the signal path reports it as a split-stack overflow, masking the real fault. There is no DATA RACE report and no hint that a foreign mapping is involved, which made this very expensive to diagnose.
What did you expect to see?
Either of:
- the access is invisible to the race detector (as happens on linux, where
mmap(NULL) placements land near 0x7f..., outside [racearenastart, racearenaend), so racecalladdr filters them out), or
- the access has valid (zero) shadow, so it is traced — and actual races on such memory become detectable.
In any case not a misleading unrecoverable runtime fatal.
Analysis
-
In race mode the heap is confined to [0x00c000000000, 0x00e000000000) (runtime/malloc.go), and TSAN shadow is mapped per heap arena (racemapshadow → __tsan_map_shadow).
-
The validity filter is one coarse interval: isvalidaddr/racecalladdr check [racearenastart, racearenaend), which is a min/max over all arenas (runtime/race.go:443 at tip, runtime/race_*.s). A foreign mapping placed between arenas passes the filter but has no shadow, so the first instrumented access faults inside __tsan_read.
-
The gap cannot be repaired from user code via __tsan_map_shadow: the Go-mode MapShadow in compiler-rt tracks a monotonic high-water mark and silently skips interior ranges (compiler-rt/lib/tsan/rtl/tsan_rtl.cpp, current main):
// Second and subsequent calls map heap.
if (shadow_end <= ctx->mapped_shadow_end)
return;
...
if (shadow_begin < ctx->mapped_shadow_end)
shadow_begin = ctx->mapped_shadow_end;
-
The same crowding also produces the separate fatal too many address space collisions for -race mode (runtime/malloc.go) when arena reservation keeps missing the window.
How this happens in the wild
In erigon (Ethereum client) CI/dev on Apple Silicon, go test -race over mdbx-heavy packages runs dozens of parallel test processes, each reserving multi-GiB memory-mapped database files. darwin's bottom-up mmap placement fills lower VA and eventually places file mappings inside the race heap window, between arenas. The result was a machine-dependent, flaky-looking fatal (split stack overflow or address space collisions) that reproduced 6/6 on affected checkouts. Working around it required hard-coding the TSAN layout (window bounds + shadow = app*2 + 0x200000000000) into the application and pre-mapping zeroed MAP_FIXED shadow over every hole in the window's shadow range at init: erigontech/erigon#21611. That works, but the layout knowledge really belongs to the runtime/TSAN, not applications.
Related issues
Possible directions
The cheapest fix appears to be in racemapshadow: when a new arena extends racearenaend (or lowers racearenastart) across a gap — i.e. the new arena is not adjacent to the current interval — also map shadow for the gap. That preserves racecalladdr's O(1) two-compare filter and makes its implicit invariant ("everything inside the interval has shadow") actually true; zero shadow is valid "no prior access" TSAN state. Alternatives: precise per-arena validity in racecalladdr (hot-path cost), or changing Go-mode MapShadow in compiler-rt to fill interior holes.
Happy to test patches on the affected hardware.
Go version
go version go1.25.7 darwin/arm64
Output of
go envin your module/workspace:macOS 15.7 (Apple M-series). The defect is present at tip by code inspection (
runtime/race.go,isvalidaddr; see analysis below).What did you do?
Ran the program below with
-race. It does three things:mmaps 1 MiB of anonymous memory with an address hint inside the race-mode heap window[0x00c000000000, 0x00e000000000). The hint only makes the placement deterministic — in real life the kernel does this unprompted once lower VA is crowded (see "How this happens in the wild" below; anonymous vs file-backed makes no difference to TSAN).racearenaendextends beyond it.Without
-raceit printsno crash; sum = 0. With-raceit dies 100% deterministically.What did you see happen?
Catching the original (non-reduced) fault under lldb shows what actually happens before the runtime mangles it:
EXC_BAD_ACCESSinside__tsan_read, at the shadow address (app*2 + 0x200000000000) of a file mapping sitting between Go heap arenas — that shadow range is unmapped. The SEGV lands whileracecallis running on the system stack, so the signal path reports it as a split-stack overflow, masking the real fault. There is noDATA RACEreport and no hint that a foreign mapping is involved, which made this very expensive to diagnose.What did you expect to see?
Either of:
mmap(NULL)placements land near0x7f..., outside[racearenastart, racearenaend), soracecalladdrfilters them out), orIn any case not a misleading unrecoverable runtime fatal.
Analysis
In race mode the heap is confined to
[0x00c000000000, 0x00e000000000)(runtime/malloc.go), and TSAN shadow is mapped per heap arena (racemapshadow→__tsan_map_shadow).The validity filter is one coarse interval:
isvalidaddr/racecalladdrcheck[racearenastart, racearenaend), which is a min/max over all arenas (runtime/race.go:443at tip,runtime/race_*.s). A foreign mapping placed between arenas passes the filter but has no shadow, so the first instrumented access faults inside__tsan_read.The gap cannot be repaired from user code via
__tsan_map_shadow: the Go-modeMapShadowin compiler-rt tracks a monotonic high-water mark and silently skips interior ranges (compiler-rt/lib/tsan/rtl/tsan_rtl.cpp, current main):The same crowding also produces the separate fatal
too many address space collisions for -race mode(runtime/malloc.go) when arena reservation keeps missing the window.How this happens in the wild
In erigon (Ethereum client) CI/dev on Apple Silicon,
go test -raceover mdbx-heavy packages runs dozens of parallel test processes, each reserving multi-GiB memory-mapped database files. darwin's bottom-upmmapplacement fills lower VA and eventually places file mappings inside the race heap window, between arenas. The result was a machine-dependent, flaky-looking fatal (split stack overfloworaddress space collisions) that reproduced 6/6 on affected checkouts. Working around it required hard-coding the TSAN layout (window bounds +shadow = app*2 + 0x200000000000) into the application and pre-mapping zeroedMAP_FIXEDshadow over every hole in the window's shadow range at init: erigontech/erigon#21611. That works, but the layout knowledge really belongs to the runtime/TSAN, not applications.Related issues
address space collisionsfatal on FreeBSD, triggered by ASLR moving mappings into the window; closed as environmental.Possible directions
The cheapest fix appears to be in
racemapshadow: when a new arena extendsracearenaend(or lowersracearenastart) across a gap — i.e. the new arena is not adjacent to the current interval — also map shadow for the gap. That preservesracecalladdr's O(1) two-compare filter and makes its implicit invariant ("everything inside the interval has shadow") actually true; zero shadow is valid "no prior access" TSAN state. Alternatives: precise per-arena validity inracecalladdr(hot-path cost), or changing Go-modeMapShadowin compiler-rt to fill interior holes.Happy to test patches on the affected hardware.