linux/arm64 nocgo OpenSSL backend: SIGSEGV in glibc start_thread when a locked OS thread is destroyed (threadentry_trampoline missing NOFRAME)

[cpuguy83]

Go version

$ go version
go version go1.26.4 linux/arm64

Microsoft Go 1.26.4, from mcr.microsoft.com/oss/go/microsoft/golang:1.26-azurelinux3.0 (OpenSSL 3.3.7, go-crypto-openssl v0.0.0-20260521135756-859040d79e1a).

Environment

Built with CGO_ENABLED=0 GOEXPERIMENT=systemcrypto,ms_nocgo_opensslcrypto, GOOS=linux GOARCH=arm64, using go1.26.4. Reproduces on native arm64 (Ampere Altra, Ubuntu 24.04) and under qemu-user emulation (docker run --platform linux/arm64 on an amd64 host), so no arm64 hardware is needed. The crash is deterministic (exit 139, typically within about 1s). Note: under qemu-user, gdb cannot ptrace the guest, so the register-level backtrace below was taken on native hardware. The faulting instructions are identical either way because it is the same binary.

What happens

With the cgo-less OpenSSL backend active, a program that starts a goroutine which calls runtime.LockOSThread() and returns without a matching runtime.UnlockOSThread() (that is, it destroys a locked OS thread) crashes with SIGSEGV. The process exits 139 with no Go panic or traceback, because the fault is inside glibc, below the Go runtime's stack. amd64 is not affected. A build with real cgo (CGO_ENABLED=1 GOEXPERIMENT=systemcrypto) is not affected.

Reproduction

package main

import (
	"crypto/sha256"
	"runtime"
	"time"
)

func main() {
	// Activate the nocgo OpenSSL backend so fakecgo is linked.
	// fakecgo sets runtime.iscgo = true, so the runtime creates Ms
	// via pthread_create. Without a crypto import the backend is not
	// linked, iscgo stays false, and the bug does not occur.
	_ = sha256.Sum256([]byte("init"))

	deadline := time.Now().Add(20 * time.Second)
	for time.Now().Before(deadline) {
		done := make(chan struct{})
		go func() {
			defer close(done)
			runtime.LockOSThread() // intentionally never unlocked
		}()
		<-done
	}
}

CGO_ENABLED=0 GOEXPERIMENT=systemcrypto,ms_nocgo_opensslcrypto GOARCH=arm64 go build -o repro .
./repro    # arm64 (native or qemu): SIGSEGV (exit 139) within ~1s, no traceback

Reproducibility and isolation (same results on native arm64 and under qemu-user)

Variant	Result
lock, no unlock, backend linked (the repro)	SIGSEGV, exit 139, deterministic
lock plus UnlockOSThread	survives (millions of teardowns)
lock, no unlock, no crypto import (backend not linked)	survives (no threadentry_trampoline in the binary)
repro with GOMAXPROCS=1 GODEBUG=asyncpreemptoff=1	SIGSEGV, exit 139
same program on amd64	survives (ran 1,323,344 teardowns, exit 0)
same program with real cgo (CGO_ENABLED=1 GOEXPERIMENT=systemcrypto)	survives

So the crash requires exactly two things: the OpenSSL backend linked (which sets iscgo=true) and a locked OS thread being destroyed. It does not require any crypto call by the worker, concurrency, or async preemption.

Observed fault (gdb, native arm64, Ubuntu 24.04 glibc)

The faulting thread is inside glibc start_thread, in its thread-exit epilogue, right after the call to the thread start routine returns (offsets are from this glibc build and are illustrative):

start_thread+368: blr  x1              ; call the start routine (threadentry_trampoline)
start_thread+376: ldr  x1, [x29, #56]  ; x29 is not the expected frame pointer
start_thread+380: str  x0, [x1, #1064] ; SIGSEGV, x1 is a wild pointer

At the point threadentry_trampoline executes RET, x29 == x30 == lr == start_thread+676 (a code address), instead of x29 holding glibc's frame pointer. AAPCS64 requires x29 to be callee-saved, and glibc relies on that across the start-routine call.

Root cause

threadentry_trampoline for arm64 is declared NOSPLIT but not NOFRAME, while it hand-manages its own frame:

// vendor/github.com/microsoft/go-crypto-openssl/internal/fakecgo/trampolines_arm64.s
TEXT threadentry_trampoline(SB), NOSPLIT, $0-0
	SUB  $(8*24), RSP
	...
	STP  (R29, R30), (8*22)(RSP)
	...

Because the function is not NOFRAME, the assembler emits automatic frame-pointer maintenance around the hand-written body. go tool objdump attributes those extra instructions to the TEXT line (line 53). With entry SP == S:

trampolines_arm64.s:53  MOVD.W R30, -16(RSP)     ; auto: SP = S-16, [S-16] = incoming LR
trampolines_arm64.s:53  MOVD   R29, -8(RSP)      ; auto: [S-24] = incoming FP (glibc's x29)
trampolines_arm64.s:53  SUB    $8, RSP, R29
trampolines_arm64.s:55  SUB    $192, RSP, RSP    ; hand-written frame
trampolines_arm64.s:61  STP    (R29, R30), 176(RSP) ; hand: [S-32] = R29, [S-24] = incoming LR
trampolines_arm64.s:70  LDP    176(RSP), (R29, R30)
trampolines_arm64.s:73  MOVD   -8(RSP), R29      ; auto: x29 <- [S-24]
trampolines_arm64.s:73  MOVD.P 16(RSP), R30
trampolines_arm64.s:73  RET

The automatic prologue saves the caller's x29 at [S-24]. The hand-written STP (R29, R30), 176(RSP) resolves its high half to that same [S-24] and overwrites it with the incoming LR. The automatic epilogue then reloads x29 from the clobbered [S-24], so the function returns with x29 == LR. glibc's next x29-relative load reads a wild pointer and faults.

How the crash path is reached

When runtime.iscgo is true (fakecgo sets it via //go:linkname _iscgo runtime.iscgo; var _iscgo = true), newosproc creates Ms with pthread_create running threadentry_trampoline. When a goroutine that locked its M exits, that M cannot be reused and is destroyed. In runtime.mexit, an M whose g0 stack was allocated by the OS (gp.stack.lo == 0, true for pthread-created Ms) takes the osStack path, which returns from mstart so the C library can free the stack and terminate the thread:

if osStack {
	// Return from mstart and let the system thread
	// library free the g0 stack and terminate the thread.
	return
}
exitThread(&mp.freeWait) // the non-pthread path; never returns

That return unwinds through threadentry_trampoline back into glibc start_thread, which is where the corrupted x29 is used.

Why amd64 is not affected

The amd64 trampoline has the same shape (not NOFRAME, hand-managed frame), but its automatic frame-pointer save and its manual save land in different slots, so there is no overlap:

trampolines_amd64.s:70  PUSHQ BP           ; auto FP save at [S-8]
trampolines_amd64.s:70  MOVQ  SP, BP
trampolines_amd64.s:72  SUBQ  $0x30, SP
trampolines_amd64.s:72  MOVQ  BP, 0x28(SP) ; manual save at [S-16], a different slot
...
trampolines_amd64.s:83  MOVQ  0x28(SP), BP ; restore from [S-16]
trampolines_amd64.s:84  POPQ  BP           ; restore glibc BP from [S-8], correct
trampolines_amd64.s:84  RET

abi_amd64.h documents the relevant discipline: such functions "should have zero frame size to suppress the automatic frame pointer." The arm64 trampoline does not follow it. Direct confirmation: the same repro built for amd64 ran 1,323,344 locked-thread teardowns and exited 0.

What it is not

• Not OpenSSL, TLS, or FIPS: the worker performs no crypto call, and the crash still occurs. The only requirement is that the backend is linked so iscgo is true.
• Not concurrency: reproduces with GOMAXPROCS=1 and a single goroutine at a time.
• Not async preemption: reproduces with GODEBUG=asyncpreemptoff=1.
• Not real cgo: the identical program built with CGO_ENABLED=1 GOEXPERIMENT=systemcrypto does not crash.

Suggested fix

Mark the arm64 threadentry_trampoline NOSPLIT|NOFRAME. That suppresses the automatic frame-pointer record, after which the hand-written STP/LDP at offset 176 correctly saves and restores the incoming x29/x30. The same file already declares this function NOSPLIT|NOFRAME for ppc64le (trampolines_ppc64le.s) and s390x (trampolines_s390x.s); arm64 is the outlier:

trampolines_ppc64le.s: TEXT threadentry_trampoline(SB), NOSPLIT|NOFRAME, $0-0
trampolines_s390x.s:   TEXT threadentry_trampoline(SB), NOSPLIT|NOFRAME, $0-0
trampolines_arm64.s:   TEXT threadentry_trampoline(SB), NOSPLIT,          $0-0

It would also be worth auditing the remaining per-arch trampolines (386, arm, riscv64, loong64) for the same construct, and adding a regression test that destroys a locked pthread-created M under the nocgo backend.

Impact

Any long-running arm64 service built with the nocgo OpenSSL backend that destroys locked OS threads crashes with exit 139 and no traceback. This includes programs that lock a goroutine to its thread for namespace or thread-affine work and let that goroutine exit.

---

Note: this root-cause analysis was produced with AI assistance (Claude Opus 4.8). The reproduction, control matrix, disassembly, and gdb capture described above were run and verified, but please independently confirm the mechanism before acting on the suggested fix.

[gdams]

Thanks for raising this bug. We've identified the required fix and will be preparing a new Go 1.26 release in the next 24-48 hours. Once we've published a new set of images I'll ping you to confirm the fix on your end.