runtime: windows/amd64 exception recovery can write below a goroutine stack and corrupt the Go heap

Go version

go version go1.27.0 windows/amd64

Output of `go env` in your module/workspace:

set AR=ar
set CC=gcc
set CGO_CFLAGS=-O2 -g
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-O2 -g
set CGO_ENABLED=0
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-O2 -g
set CXX=g++
set GCCGO=gccgo
set GO111MODULE=
set GOAMD64=v1
set GOARCH=amd64
set GOAUTH=netrc
set GOBIN=C:\Users\runneradmin\go\bin
set GOCACHE=C:\Users\runneradmin\AppData\Local\go-build
set GOCACHEPROG=
set GODEBUG=adaptivestackstart=0
set GOENV=
set GOEXE=.exe
set GOEXPERIMENT=
set GOFIPS140=off
set GOFLAGS=
set GOGCCFLAGS=-m64 -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=C:\Users\RUNNER~1\AppData\Local\Temp\go-build508021546=/tmp/go-build -gno-record-gcc-switches
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMOD=D:\a\ptah-2365-repro\ptah-2365-repro\go.mod
set GOMODCACHE=C:\Users\runneradmin\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPACKAGESDRIVER=
set GOPATH=C:\Users\runneradmin\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\hostedtoolcache\windows\go\1.27.0\x64
set GOSUMDB=sum.golang.org
set GOTELEMETRY=local
set GOTELEMETRYDIR=C:\Users\runneradmin\AppData\Roaming\go\telemetry
set GOTMPDIR=
set GOTOOLCHAIN=local
set GOTOOLDIR=C:\hostedtoolcache\windows\go\1.27.0\x64\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.27.0
set GOWORK=off
set PKG_CONFIG=pkg-config

What did you do?

On a GitHub-hosted Windows Server 2025/amd64 runner:

git clone https://github.com/denisvmedia/ptah-2365-repro
cd ptah-2365-repro

$env:GODEBUG = 'adaptivestackstart=0'
$env:GOGC = 'off'
$env:GOMEMLIMIT = 'off'
$env:GOTRACEBACK = 'system'
$env:CGO_ENABLED = '0'
$env:GOAMD64 = 'v1'
$env:GOTOOLCHAIN = 'local'

go build -trimpath -buildvcs=false -o repro.exe .
.\repro.exe panic 2> panic.log
.\repro.exe fault 2> fault.log

The panic mode is a negative control. Both modes use the same goroutine stack size, depth, heap adjacency, deferred recovery, and explicit post-recovery GC. The only difference is an ordinary Go panic versus a real read access violation at address 0x118.

The hosted pool is heterogeneous. In the instrumented four-job run, the exact failure occurred on the sole Intel/AMX host; the three AMD/non-AMX jobs were clean. Later uninstrumented runs reproduce intermittently on the same pool. The repository workflow runs four attempts and validates either result fail-closed.

Standalone source and workflow:
https://github.com/denisvmedia/ptah-2365-repro/tree/72cb4a8a27ef8c86347f8eee237c23e7b6681e85

What did you see happen?

The panic control exits 0 and the 32 bytes preceding stack.lo are unchanged:

REPRO_TAIL before=0000000000000000000000000000000000000000000000000000000000000004 after=0000000000000000000000000000000000000000000000000000000000000004
REPRO_RESULT clean

On an affected host the fault mode first records a write below stack.lo, then the next GC aborts:

REPRO_PAIR span=11771934179328 stack_lo=11771934187520
REPRO_TAIL before=0000000000000000000000000000000000000000000000000000000000000004 after=17c5ee29f77f000048a0f4ddb40a0000f192ec29f77f000020a0f4ddb40a0000
runtime: marked free object in span ..., elemsize=16 freeindex=0 (...)
...
fatal error: found pointer to free object
runtime.(*mspan).reportZombies(...)

panic exits 0; fault exits 2. The runtime report contains exactly 496 contiguous size-class-16 rows, starting at the REPRO_PAIR predecessor span. The witness and go env above are from run 33366178316, attempt 1.

What did you expect to see?

The read access violation should become a recoverable Go runtime panic without writing outside the goroutine stack. The ordinary panic and fault modes should both leave the adjacent heap page unchanged and exit normally after recovery.

Analysis

Go 1.27 reserves an extra 4096 bytes for goroutine stacks on Windows:

stackSystem = goos.IsWindows*4096 + ...

The failing GitHub runner was Windows Server 2025. On another failing Intel host, XSTATE measurements showed that enabling the AMX XTILEDATA component increases the Windows context size by 8192 bytes:

full context:          11567 bytes
without XTILEDATA:      3375 bytes
difference:             8192 bytes

On AMD/non-AMX hosts, the required context was much smaller and the reproducer stayed clean. This is a strong size correlation, although it does not prove that Windows writes the entire 8192-byte XTILEDATA area during exception handling. The measurement is in run 33350325788.

The reproducer positions the faulting instruction near the bottom of a goroutine stack. After the access violation is recovered, the last 32 bytes immediately below stack.lo have changed. The written values resolve to runtime stack/exception-handling addresses, including:

runtime.systemstack+0x37
stack.lo+0x48
runtime.sigtrampgo+0x91
stack.lo+0x20

This shows that the exception path writes at least 32 bytes below the goroutine stack before GC runs.

The memory directly below the stack is intentionally a live 8 KiB size-class-16 heap span. Under Green Tea GC, its mark metadata occupies that area. The overwrite corrupts those mark bits, and the next GC reports found pointer to free object. The GC detects the corruption; it does not appear to cause it.

The reproducer does not create this with a dangling pointer. unsafe is used only to arrange the memory layout and inspect it. The same memory remains unchanged in an otherwise equivalent panic/recover control and changes only after the hardware access violation.

This is also separate from golang/go#80799. Stock Go already produces both the memory overwrite and the runtime abort. #80799 only affects how reportZombies prints the already-detected corruption.

This looks similar to golang/go#70288, where Windows exception handling required a larger system-stack reserve. The current evidence suggests that the fixed 4096-byte Windows reserve may be too small on hosts where AMX XTILEDATA is enabled. It does not yet prove exactly which Windows/runtime component performs the overwrite or whether increasing stackSystem alone is the complete fix.

Related Issues

runtime: intermittent ACCESS_VIOLATION in (*unwinder).next during shrinkstack on Windows amd64 (Go 1.26.0) #77955 (closed)
cmd/asm: amd64 PCALIGN between labels generates wrong jump targets #74648 (closed)
runtime: access violation in gentraceback on windows/386 #50144
runtime: Fatal Error in heapBitsForObject during GC on Windows #24768 (closed)
runtime: "fatal error: unexpected signal" 0xC0000005 on Windows for a small program with a large allocation #37470 (closed)
runtime: leftover panic in traceback #8132 (closed)
internal/poll: access violation (0xc0000005) in (*FD).execIO deferred cleanup during concurrent UDP ReadFrom/WriteTo teardown on windows/amd64 #80195 (closed)
runtime: intermittent startup panics across multiple Go applications on Windows 11 #79249
runtime: reportZombies throws "found pointer to free object" in sweeper #71451 (closed)
runtime: windows/386 SEH corruption #7470 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

Change https://go.dev/cl/824724 mentions this issue: runtime: crash if a Windows exception is dispatched below the goroutine stack

Thanks for reporting this issue, and for the reproducer. I've submitted https://go.dev/cl/824724 to turn the heal corruption into a deterministic panic.

On the other hand, I'm afraid the current design can't cope with such big CONTEXTs. We can't keep growing the system-reserved goroutine stack indefinitely. The long term fix is to stop using Vectored Exception Handling (VEH) and use Structured Error Handling (SEH) instead, tracked in #47576, given that SEH don't need a dedicated stack where to dispatch exceptions,

@qmuntal

Thanks @qmuntal. I agree SEH may be the right long-term direction, but I tested the exact CL 824724 patch set 1, and it does not catch this reproducer.

The repro repo now includes the exact patch and a permanent stock-vs-patched A/B workflow:
https://github.com/denisvmedia/ptah-2365-repro

The result was:

MISSED_GUARD STOCK=EXACT PATCHED=EXACT

https://github.com/denisvmedia/ptah-2365-repro/actions/runs/33403553197/job/99525466135

On the same Intel Xeon Platinum 8573C runner, both stock and patched fault runs:

exited 2;
overwrote the same 32 bytes below stack.lo, including pointers at stack.lo+0x48 and stack.lo+0x20;
later failed in runtime.(*mspan).reportZombies.

The patched run did not emit exception dispatched below goroutine stack bottom.

The workflow also verifies the exact source blobs and confirms the new diagnostic string is present in the patched binary, so this is not a stale or missing overlay. A second run reproduced the same MISSED_GUARD result twice:
https://github.com/denisvmedia/ptah-2365-repro/actions/runs/33403988570

REPRO_TAIL records the overwrite before GC and before the #80799 reporting path, so this still appears to be actual memory corruption rather than just a reporter failure.

It looks like sys.GetCallerSP() is still above gp.stack.lo when the new check runs, even though the exception path subsequently writes below stack.lo.

So CL 824724 may still be useful as defense in depth, but it does not catch this failure. Could we keep this issue open?

cc @golang/runtime

feat(server): surface unclean exits on the attention panel and exit o…

Merge pull request #103 from rodrigorjsf/ci/102-windows-go-runtime-retry

fix(review): U1 retry the Windows traceback-unwind runtime crash

logging: panic explicitly in the typed-nil error fixture

Another data point, from an ordinary test suite with no unsafe, no cgo and no SetPanicOnFault. A plain nil-pointer write recovered with recover() is enough to hit this on AMX hosts. It shows up as several different runtime fatals, among them invalid g status.

Go version

go version go1.27.1 windows/amd64 (GitHub-hosted windows-latest, Windows Server 2025, go test -race)

What did you do?

Ran the root package tests of https://github.com/junioryono/godi (commit ec70c42) with go test -race ./... on windows-latest. The suite has one test that takes a hardware fault, TestLifetime/NilPointerUnmarshal. It is a parallel subtest that calls a pointer-receiver method on a nil pointer inside testify's assert.Panics:

var ptr *Lifetime
assert.Panics(t, func() { _ = ptr.UnmarshalText([]byte("Singleton")) }) // *sl = Singleton on nil

Our CI crashed once in about 80 runs. To reproduce it, we looped the prebuilt race test binary on many fresh windows-latest runners and recorded each runner's CPU:

Runner CPU	Jobs	Full-suite runs	Crashed runs
Intel Xeon Platinum 8573C (AMX)	1	150	10
Intel Xeon Platinum 8573C (AMX)	1	100 (+100 with the faulting test skipped)	0
AMD EPYC 7763 / 9V74 / 9V45	23	2,300 (+2,300 skipped)	0
CPU not recorded (first round; race, no-race and go1.26 mixed)	16	1,730	0
macOS arm64 (local)	—	300	0

The affected runner was 8573C job 1. On it, running only -test.run=TestLifetime/NilPointerUnmarshal -test.count=500 also killed the process: 2 of the first 4 runs died silently, with no output and a non-zero exit. The job hit its time limit before the "faulting test skipped" control and the standalone loops ran on that host. The second 8573C job ran clean, which matches the per-job variance reported above (73 of 162 jobs crashing on that model).

What did you expect to see?

The nil dereference becomes a recoverable runtime panic, and the test passes, as it does on every non-AMX host.

What did you see happen?

The 10 crashes on the AMX host report corruption in assorted runtime structures:

fatal error: invalid g status                                     (x2)
fatal error: newproc1: new g is not Gdead
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)
fatal error: send on synctest channel from outside bubble
fatal error: unexpected signal during runtime execution           (gfget: corrupted free-g list)
unexpected fault address 0xffffffffffffffff
unexpected fault address 0xc1403a489f
<process exits with no output>                                    (x2)

The original CI crash (invalid g status):

runtime:   gp: gp=0xc00077ed20, goid=0, gp->atomicstatus=0
runtime: getg:  g=0xc00043c960, goid=0,  g->atomicstatus=0
fatal error: invalid g status

runtime stack:
runtime.throw({0x14067dc65?, 0x4adffc78?})
	runtime/panic.go:1243 +0x4d
runtime.suspendG(0xc00077ed20)
	runtime/preempt.go:136 +0x50a
runtime.markroot.func1()
	runtime/mgcmark.go:298 +0x85
runtime.markroot(0xc00003b260, 0x161, 0x0)
	runtime/mgcmark.go:280 +0x1a5
runtime.gcDrainN(0xc00003b260, 0x78c8)
	runtime/mgcmark.go:1441 +0x1a5
runtime.gcAssistAlloc1(0xc0002f8d20, 0x78c8)
	runtime/mgcmark.go:745 +0xea
...
goroutine 140713693832584 gp=0xc00077eb40 m=5369256288 mp=0x140a71240 [??? (scan), locked to thread]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal 0xc0000005 code=0x1 addr=0x140a7135c pc=0x140074779]

The g beside the one being scanned (0xc00077eb40) holds what look like register values rather than goroutine state. We rebuilt the same binary on the same toolchain and checked these values against it:

goid 0x7ffa75bc6588 is a system-DLL address.
m 0x140085960 is the entry of runtime.initMetrics.func10.
mp 0x140a71240 points into go:funcdesc.

One of the AMX crashes is in a sibling parallel subtest, TestLifetime/ConcurrentAccess, which starts goroutines right after the nil-write subtest:

fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x14005a0c5]
runtime.(*gList).pop(...)            runtime/proc.go:7904
runtime.gfget(0xc000042008)          runtime/proc.go:5567
runtime.newproc1(...)                runtime/proc.go:5359
...
sync.(*WaitGroup).Go(...)            sync/waitgroup.go:238
github.com/junioryono/godi/v6.TestLifetime.func12(...)   lifetime_test.go:213

Logs:

Original CI failure: https://github.com/junioryono/godi/actions/runs/37255239968/job/111590779212
Reproduction (AMX host is shard 26, crash logs in the crashes-26 artifact): https://github.com/junioryono/godi/actions/runs/37258698130
Non-AMX control run: https://github.com/junioryono/godi/actions/runs/37257707581

Independent A/B validation of CL 828724 patch set 1 (cf0ed477) against its parent 38d1265e (tip, which already has CL 824724). This used a different workload from the one in the CL description.

Setup. Both toolchains were built from source on windows-latest (Windows Server 2025, 10.0.26100). Then 60 fresh runners each ran both builds on the same host, alternating stock and patched every round:

godi-nil: the godi race test binary from my earlier comment, -test.run=TestLifetime/NilPointerUnmarshal -test.count=500 (a plain nil-pointer write recovered by recover)
godi-suite: the full godi race suite, -test.count=1
faultstress: the SetPanicOnFault reproducer from this thread, -test.count=20

AMX hosts looped for 40 minutes. Other hosts ran 3 rounds.

Runner CPU	`InitializeContext(CONTEXT_ALL\|CONTEXT_XSTATE)`	patched reserve / min stack	jobs	stock failed / runs	patched failed / runs
Intel Xeon 6973P-C	11567	13616 / 16384	3	2949 / 2949	0 / 2949
Intel Xeon Platinum 8573C	11567	13616 / 16384	1	894 / 894	0 / 894
Intel Xeon Platinum 8573C	3247	5296 / 8192	1	0 / 9	0 / 9
Intel Xeon Platinum 8370C	3247	5296 / 8192	1	0 / 9	0 / 9
AMD EPYC 9V45 / 9V74	3263	5312 / 8192	21	0 / 189	0 / 189
AMD EPYC 7763 / 9V74	1663	4096 / 8192	33	0 / 297	0 / 297

On the hosts with an 11,567-byte context, every stock run died. 3840 runs died with fatal error: exception dispatched below goroutine stack bottom, and 3 hung after it, for example a second fatal error: stack not a power of 2 from gfget. All three workloads failed equally. The patched build ran 3843 times on the same hosts without a single failure. A dry run on another 8573C (context 11567) gave the same result: stock failed 81/81, patched 0/81.

A few other observations:

One 8573C reported a 3,247-byte context, so the AMX tile state wasn't enabled there. Both builds stayed clean on it. That likely explains the per-host variance on that model reported above, including our clean 8573C job.
TestExceptionStackReserve passed on all 60 hosts. With the patch, go test -short runtime passed on all four AMX hosts, and so did go test -short -race -run 'Stack|Exception|Fault|Panic|Signal|Goroutine' runtime.
Since stackGuard becomes a variable on every port, I also built runtime and its test binary for every go tool dist list target. Everything compiled, apart from targets that require cgo linking, which were skipped. go test -short runtime runtime/debug passes on darwin/arm64.
A side note on stock tip: after the CL 824724 throw, the traceback sometimes fails again with fatal error: unsafe.Slice: len out of range / panic during panic in tracebackHexdump, because the sigtramp frame is below stack.lo. With this CL that path isn't reached.

Workflow, scripts and per-host logs/CSVs: https://github.com/junioryono/go81238-amx-validation (run 37266360678).

client/thin, core/pki: return "<nil>" from String on a nil receiver

plumbing: transport, Render nil origins without a fault

client/thin, core/pki: return "<nil>" from String on a nil receiver

fix: no recovered nil dereference on each Telegram API call

fix: no recovered nil dereference on each Telegram API call (#30)

client/thin, core/pki: return "<nil>" from String on a nil receiver

security: wipe dropped secrets, bound the thin-client listener, keep …

fix(jobs): check River client settings instead of recovering panics

test(windows): skip nil-deref panic recovery tests

Navigation Menu

Repository navigation

runtime: windows/amd64 exception recovery can write below a goroutine stack and corrupt the Go heap #81238

Go version

Output of `go env` in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Analysis

35 remaining items

Go version

What did you do?

What did you expect to see?

What did you see happen?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

runtime: windows/amd64 exception recovery can write below a goroutine stack and corrupt the Go heap #81238

Description

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

Activity

denisvmedia commented on Aug 31, 2026

Analysis

gabyhelp commented on Aug 31, 2026

gopherbot commented on Aug 31, 2026

qmuntal commented on Aug 31, 2026

denisvmedia commented on Sep 1, 2026

prattmic commented on Sep 4, 2026

35 remaining items

junioryono commented on Oct 5, 2026

Go version

What did you do?

What did you expect to see?

What did you see happen?

junioryono commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Output of `go env` in your module/workspace: