runtime: fatal error: semasleep on Darwin signal stack

[oarkflow]

Go version

go version go1.25.0 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/user/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/user/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/k7/7hz0_zdj74g3jdv17dznvh640000gn/T/go-build1933108180=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOOS='darwin'
GOPATH='/Users/user/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/user/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

func RecoverPanic(labelGenerator func() string) {
	if r := recover(); r != nil {
		defer func() {
			if rr := recover(); rr != nil {
				// If logging or labelGenerator panics, just print a minimal message
				fmt.Printf("[PANIC] - error during panic recovery: %v\n", rr)
			}
		}()
		pc, file, line, ok := runtime.Caller(2)
		funcName := "unknown"
		if ok {
			fn := runtime.FuncForPC(pc)
			if fn != nil {
				funcName = fn.Name()
			}
		}
		log.Printf("[PANIC] - recovered from panic in %s (%s:%d): %v\nStack trace: %s", funcName, file, line, r, debug.Stack())
	}
}

This function was included in most of the functions recovering panic

What did you see happen?

In random, getting following error:

fatal error: semasleep on Darwin signal stack
panic during panic

goroutine 0 gp=0x14000602000 m=10 mp=0x14000600008 [idle]:
runtime.throw({0x1053635db?, 0x1400060b8f8?})
	/usr/local/go/src/runtime/panic.go:1094 +0x34 fp=0x1400060b8b0 sp=0x1400060b880 pc=0x104182274
runtime.semasleep(0xffffffffffffffff)
	/usr/local/go/src/runtime/os_darwin.go:49 +0x168 fp=0x1400060b910 sp=0x1400060b8b0 pc=0x104148ef8
runtime.lock2(0x106b4b9f0)
	/usr/local/go/src/runtime/lock_spinbit.go:250 +0x37c fp=0x1400060b970 sp=0x1400060b910 pc=0x10412217c
runtime.lockWithRank(...)
	/usr/local/go/src/runtime/lockrank_off.go:24
runtime.lock(...)
	/usr/local/go/src/runtime/lock_spinbit.go:152
runtime.startpanic_m()
	/usr/local/go/src/runtime/panic.go:1376 +0xd0 fp=0x1400060b9a0 sp=0x1400060b970 pc=0x10414bbe0
runtime.sighandler(0x6, 0x14000600008?, 0x1400060ba40?, 0x140006021c0)
	/usr/local/go/src/runtime/signal_unix.go:759 +0x288 fp=0x1400060ba10 sp=0x1400060b9a0 pc=0x104165928
runtime.sigtrampgo(0x6, 0x1400060bbb0, 0x1400060bc18)
	/usr/local/go/src/runtime/signal_unix.go:490 +0x108 fp=0x1400060ba90 sp=0x1400060ba10 pc=0x1041651a8
runtime.sigtrampgo(0x6, 0x1400060bbb0, 0x1400060bc18)
	<autogenerated>:1 +0x1c fp=0x1400060bac0 sp=0x1400060ba90 pc=0x10418caac
runtime.sigtramp()
	/usr/local/go/src/runtime/sys_darwin_arm64.s:227 +0x4c fp=0x1400060bb80 sp=0x1400060bac0 pc=0x10418b65c

goroutine 328 gp=0x14000fdafc0 m=10fatal error: semasleep on Darwin signal stack mp=0x14000600008fatal error: semawakeup on Darwin signal stack
stack trace unavailable

panic during panic

What did you expect to see?

There should not be any error.

[Jorropo]

cc @golang/runtime

[gabyhelp]

Related Issues

• runtime: stack grow panic tracing back through sigpanic from signal handler #23484
• runtime: unexpected return pc for runtime.sigtramp on darwin/arm64 builder #29305 (closed)
• runtime: fatal error: unexpected signal during runtime execution #60783 (closed)
• "panic during panic" on arm64 #34933 (closed)
• runtime #60196 (closed)
• go/build: Segfault on Go 1.6.2 darwin/amd64 #17912 (closed)
• runtime: Build interrupted with panic and segmentation fault on OS X Sierra #16669 (closed)
• runtime: endless sigtramp in the stack of coredump generated by panic on arm64 #58998
• runtime: unexpected return pc for runtime.sigpanic on darwin in x/sys/unix #47776 (closed)
• panic: runtime: frame runtime.memeq untyped args #10678 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

The crash occurs when attempting to lock paniclk in startpanic_m.

It's odd to take a lock in the signal handler, but semantically I believe it is fine. It is guarded by a check and increment of gp.m.dying which prevents the same thread from attempting acquiring the lock.

On Linux, it is completely fine, as lock -> semasleep is implemented with direct futex calls.

On Darwin, semasleep is implemented with pthread mutexes. Since those aren't considered async-signal-safe, we explicitly throw if reached from the signal handler.

Thus, contention on paniclk on Darwin may hit this throw.

I'm not sure what this has to do with the RecoverPanic snippet. paniclk is only taken for fatal panics. Recovered panics are not fatal, so they shouldn't touch this lock. I'd expect to see this issue randomly with many goroutines concurrently triggering a fatal (not recovered) panic.

Maybe paniclk should just be a simple spinlock?

[mknyszek]

@prattmic Maybe we could have an asyncSignalSafeMutex type which is just a regular runtime mutex on platforms where it's fine, and a spinlock everywhere else.

We definitely have other use-cases for an async-signal-safe mutex. IIRC, we hack together a(n untyped) spinlock for CPU profile samples and CPU profile events written to execution traces. It would be nice if that just looked like a regular lock.

We could also enforce that all such locks are leaf locks in the static-lock-ranking lock order, and possibly add some other checks, like calling out an order violation if the lock is grabbed outside of a signal and signals aren't masked.

[ianlancetaylor]

It seems like it would also be fine to just not bother with the signal crash, since the program is crashing anyhow. We could change semasleep on Darwin so that if it is running on the signal stack, it just does an infinite sleep. Or maybe sleep for 1 second and then crash.