runtime: SIGSEGV in gcStart on Bottlerocket 1.65.0 / Linux 6.12.103 — regression vs go1.25.14

What version of Go are you using (`go version`)?

Crashes:

go1.26.8

Does not crash — same source, same environment, only the build toolchain changed:

go1.25.14

What operating system and processor architecture are you using (`go env GOOS GOARCH`)?

GOOS=linux
GOARCH=amd64

Container OS: Bottlerocket OS 1.65.0 (aws-k8s-1.35), containerd 2.2.7, cgroup v2.
Host kernel: 6.12.103 — crashes.
Same container image, same binary, run on a host kernel 6.12.94 (Bottlerocket 1.64.0, otherwise
identical AMI family) — does not crash. This is the only environment variable that reproducibly
flips the outcome (see "Isolation" below).

Reproduced on AWS EC2 (all amd64, all crash on kernel 6.12.103): c8i-flex.2xlarge (Intel),
m7i-flex.2xlarge (Intel), c6a.2xlarge (AMD) — so it isn't CPU-vendor-specific.

What did you do?

A small, otherwise-idle internal Go service (a Kubernetes-in-cluster HTTP API server, low request
volume) started crash-looping in production after its EKS nodegroup rolled from Bottlerocket 1.64.0
(kernel 6.12.94) to 1.65.0 (kernel 6.12.103). Isolated it with a controlled A/B, one variable changed
at a time, using throwaway Kubernetes pods pinned to specific nodes via nodeName:

Same image, 3 different nodes spanning Intel and AMD instance families, all on kernel 6.12.103
→ crashes every time, ruling out CPU vendor/family.
Same image, a node on kernel 6.12.94 (older Bottlerocket) → clean for 5.5+ minutes (well past
every crash window seen elsewhere), ruling out resource limits / GOMAXPROCS / anything about the
binary itself and isolating the kernel version.
Rebuilt the identical source with golang:1.25-alpine instead of golang:1.26-alpine as the
Docker build base, ran it on the exact node that had just crashed twice under 1.26 → clean for
5.5+ minutes.

Build command (both variants):

CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /app .

No GOMAXPROCS/GOMEMLIMIT env vars set; no CGO; a small program that does routine
crypto/x509/net/http work (parses a CA bundle into an x509.CertPool via AppendCertsFromPEM
at startup for a couple of TLS clients) and otherwise sits mostly idle serving occasional HTTP
requests.

The crash is not tied to one code path — I caught it three separate times, twice via completely
different triggers:

Crash A — idle, ~2 minutes after start (matches Go's forced-GC timer):

SIGSEGV: segmentation violation
PC=0x42b4c4 m=3 sigcode=1 addr=0x8

goroutine 0 gp=0x2167b704f2c0 m=3 mp=0x2167b70c3008 [idle]:
runtime.gcStart.func4()
	runtime/mgc.go:949 +0x44 fp=0x2167b70d9fc8 sp=0x2167b70d9f80 pc=0x42b4c4
runtime.systemstack(0x0)
	runtime/asm_amd64.s:562 +0x4a fp=0x2167b70d9fd8 sp=0x2167b70d9fc8 pc=0x489bea

goroutine 1 gp=0x2167b704e1e0 m=nil [select]:
runtime.gopark(0x2167b721fda0?, 0x2?, 0x0?, 0x0?, 0x2167b721fd6c?)
	runtime/proc.go:462 +0xce fp=0x2167b7697c00 sp=0x2167b7697be0 pc=0x4841ce
runtime.selectgo(0x2167b7697da0, 0x2167b721fd68, 0x2?, 0x0, 0x1?, 0x1)
	runtime/select.go:351 +0xaa5 fp=0x2167b7697d30 sp=0x2167b7697c00 pc=0x4626a5
main.run(...)
main.main()
runtime.main()
	runtime/proc.go:290 +0x2d5

(full goroutine dump available if useful; trimmed here to the relevant frames)

Crash B — immediately at startup, triggered by an ordinary allocation during
x509.CertPool.AppendCertsFromPEM → parseCertificate → parsing an ASN.1 big-int for a
certificate's public key:

SIGSEGV: segmentation violation
PC=0x42b4c4 m=6 sigcode=1 addr=0x8

goroutine 0 gp=0x2167b71e7860 m=6 mp=0x2167b6a2008 [idle]:
runtime.gcStart.func4()
	runtime/mgc.go:949 +0x44 fp=0x2167b1c27fc8 sp=0x2167b1c27f80 pc=0x42b4c4
runtime.systemstack(0x0)
	runtime/asm_amd64.s:562 +0x4a

goroutine 69 gp=0x2167b1c05680 m=6 mp=0x2167b1c00808 [running]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:516 +0x8
runtime.gcStart({0x7fbce84044e8?, 0x5a6d37?, 0xe1f8d8e6?})
	runtime/mgc.go:947 +0x651
runtime.mallocgcSmallNoscan(0x3?, 0xc?, 0x74?)
	runtime/malloc.go:1448 +0x28f
runtime.mallocgc(0x228, 0x8f72c0, 0x1)
	runtime/malloc.go:1143 +0xcc
runtime.makeslice(0x40?, 0x611722823566?, 0x0?)
	runtime/slice.go:117 +0x49
math/big.nat.make(...)
	math/big/nat.go:67
math/big.nat.setBytes(...)
	math/big/nat.go:1023 +0x99
math/big.(*Int).SetBytes(...)
	math/big/int.go:519
vendor/golang.org/x/crypto/cryptobyte.(*String).readASN1BigInt(...)
	vendor/golang.org/x/crypto/cryptobyte/asn1.go:330 +0x105
vendor/golang.org/x/crypto/cryptobyte.(*String).ReadASN1Integer(...)
	vendor/golang.org/x/crypto/cryptobyte/asn1.go:290 +0x3ff
crypto/x509.parsePublicKey(...)
	crypto/x509/parser.go:266 +0x1e9
crypto/x509.parseCertificate(...)
	crypto/x509/parser.go:1011 +0x9d9
crypto/x509.ParseCertificate(...)
	crypto/x509/parser.go:1077 +0x1d
crypto/x509.(*CertPool).AppendCertsFromPEM(...)
	crypto/x509/cert_pool.go:231 +0x125

Crash C — same as B but a different ASN.1 field (ReadASN1ObjectIdentifier instead of the
big-int path), on a different node, confirming it's not one specific allocation size/shape:

SIGSEGV: segmentation violation
PC=0x42b4c4 m=6 sigcode=1 addr=0x8
...
runtime.gcStart.func4()
	runtime/mgc.go:949 +0x44
...
vendor/golang.org/x/crypto/cryptobyte.(*String).ReadASN1ObjectIdentifier(...)
	vendor/golang.org/x/crypto/cryptobyte/asn1.go:461 +0x65
crypto/x509.parseAI(...)
	crypto/x509/parser.go:185 +0x8b
crypto/x509.parseCertificate(...)
	crypto/x509/parser.go:952 +0x408

All three: identical PC=0x42b4c4 (inside runtime.gcStart.func4, runtime/mgc.go:949) and
identical faulting address addr=0x8 (a nil pointer + 8-byte offset), regardless of what triggered
the GC start or which physical node/CPU it ran on — only the Go toolchain version and the kernel
version change the outcome.

What did you expect to see?

No crash — gcStart running to completion regardless of what triggered it, as it does under
go1.25.14 in the identical environment.

What did you see instead?

A deterministic SIGSEGV inside the Go runtime's own GC-start path, with no application-level
frame anywhere near the fault — reproduced on 3 different EC2 instance types/CPU vendors, always
on the same Bottlerocket 1.65.0 (kernel 6.12.103) build, never on Bottlerocket 1.64.0
(kernel 6.12.94) or with go1.25.14 on the exact same 6.12.103 host.

Additional notes

I did not find an existing issue matching this exact signature. The closest related report I found
is #77731 (a different 1.26 regression: a kernel/futex-facing runtime change broke amd64→mips-style
old-kernel compatibility, fixed via a 1.26.1 backport) — different architecture and the opposite
kernel-age direction from what I'm seeing, but it suggests 1.26 touched kernel-facing runtime code
in ways that have already produced at least one other kernel-version-sensitive regression.

Related Issues

runtime: segment violation in mexit/vgetrandomPutState/mallocgcSmallNoscan #73577 (closed)
runtime: SIGSEGV in buildTypeAssertCache during crypto/tls TLS 1.3 client handshake #80933
runtime: segmentation fault from `vgetrandomPutState` and `runtime.growslice` w/ `runtime.OSLockThread` #73141 (closed)
runtime: SIGSEGV crash on Linux in docker container, possible net/http crash #51113
runtime: segmentation violation when debug and using race detector #77597 (closed)
crypto/x509: crash on macOS Tahoe 26 #75684 (closed)
runtime: SIGSEGV on nil pointer in mheap.freeManual #73628 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

@prattmic

cc @prattmic

Could you disassemble your binary and include the assembly for runtime.gcStart.func4, which should include the faulting PC?

The line with the fault does not contain any obvious dereference.

Thanks. Here is runtime.gcStart.func4 from the binary that crashed. It was built with -ldflags='-s -w', so I rebuilt the same source with the same toolchain (go1.26.8, CGO_ENABLED=0 -trimpath). The stripped rebuild is bit-identical to the crashing binary (same sha256), and the .text of the symbol-bearing rebuild is byte-identical, so these addresses match the crash:

TEXT runtime.gcStart.func4(SB) runtime/mgc.go
  mgc.go:947		0x42b480		493b6610		CMPQ SP, 0x10(R14)
  mgc.go:947		0x42b484		766b			JBE 0x42b4f1
  mgc.go:947		0x42b486		55			PUSHQ BP
  mgc.go:947		0x42b487		4889e5			MOVQ SP, BP
  mgc.go:947		0x42b48a		4883ec38		SUBQ $0x38, SP
  mgc.go:947		0x42b48e		4c8b4210		MOVQ 0x10(DX), R8
  mgc.go:947		0x42b492		4c89442428		MOVQ R8, 0x28(SP)
  mgc.go:947		0x42b497		488b5208		MOVQ 0x8(DX), DX
  mgc.go:947		0x42b49b		4889542430		MOVQ DX, 0x30(SP)
  mgc.go:948		0x42b4a0		410fb618		MOVZX 0(R8), BX
  mgc.go:948		0x42b4a4		498b4808		MOVQ 0x8(R8), CX
  mgc.go:948		0x42b4a8		498b7810		MOVQ 0x10(R8), DI
  mgc.go:948		0x42b4ac		498b7018		MOVQ 0x18(R8), SI
  mgc.go:948		0x42b4b0		31c0			XORL AX, AX
  mgc.go:948		0x42b4b2		e8c9700200		CALL runtime.startTheWorldWithSema(SB)
  mgc.go:948		0x42b4b7		488b542430		MOVQ 0x30(SP), DX
  mgc.go:948		0x42b4bc		488902			MOVQ AX, 0(DX)
  mgc.go:949		0x42b4bf		4c8b442428		MOVQ 0x28(SP), R8
  mgc.go:949		0x42b4c4		4d8b4008		MOVQ 0x8(R8), R8        <-- faulting PC
  mgc.go:949		0x42b4c8		4c29c0			SUBQ R8, AX
  mgc.go:949		0x42b4cb		480105268fa800		ADDQ AX, runtime.work+696(SB)
  mgc.go:950		0x42b4d2		4c8b02			MOVQ 0(DX), R8
  mgc.go:950		0x42b4d5		4c8905048fa800		MOVQ R8, runtime.work+672(SB)
  mgc.go:953		0x42b4dc		488b1a			MOVQ 0(DX), BX
  mgc.go:953		0x42b4df		488d059a9eaa00		LEAQ runtime.gcCPULimiter(SB), AX
  mgc.go:953		0x42b4e6		e8b5300000		CALL runtime.(*gcCPULimiterState).finishGCTransition(SB)
  mgc.go:954		0x42b4eb		4883c438		ADDQ $0x38, SP
  mgc.go:954		0x42b4ef		5d			POPQ BP
  mgc.go:954		0x42b4f0		c3			RET
  mgc.go:947		0x42b4f1		e88ae70500		CALL runtime.morestack.abi0(SB)
  mgc.go:947		0x42b4f6		eb88			JMP runtime.gcStart.func4(SB)

So the fault is the reload of the captured stw pointer from its spill slot 0x28(SP) after startTheWorldWithSema returns. The same pointer was spilled at 0x42b492 and dereferenced without trouble at 0x42b4a0–0x42b4ac, before the call. The register dump from the first crash agrees:

rax    0xb70beec8
rbx    0x2
rcx    0x2
rdx    0x2167b70beee8
rdi    0x2167b7101160
rsi    0x81
rbp    0x2167b70d9fb8
rsp    0x2167b70d9f80
r8     0x0
r9     0x0
r10    0x0
r11    0x216
r12    0x2167b70d9e70
r13    0x0
r14    0x2167b704f2c0
r15    0x0
rip    0x42b4c4
rflags 0x10202

rsp matches the gcStart.func4 frame's sp in the traceback. r8 == 0 is what was reloaded from 0x28(SP). The neighboring slot 0x30(SP) survived the same call: it was reloaded into rdx (non-nil) at 0x42b4b7, and the store through it at 0x42b4bc succeeded. So one 8-byte slot in this g0 frame (rsp+0x28) seems to be zeroed while startTheWorldWithSema runs. All three crashes have the same PC and addr=0x8.

Does your distribution use uretprobes? Those are incompatible with Go due to the way they attempt to modify the return address on the stack (#22008 / iovisor/bcc#1320). I'm not sure they'd cause exactly this failure mode, but it sounds plausible.

Navigation Menu

Repository navigation

runtime: SIGSEGV in gcStart on Bottlerocket 1.65.0 / Linux 6.12.103 — regression vs go1.25.14 #81559

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env GOOS GOARCH`)?

What did you do?

What did you expect to see?

What did you see instead?

Additional notes

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

runtime: SIGSEGV in gcStart on Bottlerocket 1.65.0 / Linux 6.12.103 — regression vs go1.25.14 #81559

Description

What version of Go are you using (go version)?

What operating system and processor architecture are you using (go env GOOS GOARCH)?

What did you do?

What did you expect to see?

What did you see instead?

Additional notes

Activity

gabyhelp commented on Sep 16, 2026

mrkfrmn commented on Sep 22, 2026

prattmic commented on Sep 22, 2026

DanOfir commented on Sep 23, 2026

prattmic commented on Sep 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env GOOS GOARCH`)?