runtime: signal 27 received on thread with no signal stack

[ukai]

Go version

go version go1.25.7 darwin/arm64

Output of go env in your module/workspace:

AR='ar'
CC='clang'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='clang++'
GCCGO='gccgo'
GO111MODULE=''
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/Users/ukai/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/ukai/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/fg/y90rmtzj3hz3hqtpwn0qwzmc000p7r/T/go-build960639305=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD='/Users/ukai/src/crbuild/build/siso/go.mod'
GOMODCACHE='/Users/ukai/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ukai/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/ukai/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.25.7'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

go binary (without cgo) runs with profilers

$ go version -m third_party/siso/cipd/siso
...
        build   -buildmode=exe
        build   -compiler=gc
        build   -trimpath=true
        build   CGO_ENABLED=0
        build   GOARCH=arm64
        build   GOOS=darwin
        build   GOARM64=v8.0

https://ci.chromium.org/ui/p/chromium/builders/build/ios-build-perf-developer/7889/overview
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8691654715001248849/+/u/Clean_builds/Build_chrome_without_remote_cache_with_Siso_in_native_mode/stdout

https://ci.chromium.org/ui/p/chromium/builders/build/ios-build-perf-siso/25669/overview
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8690003051682517681/+/u/Clean_builds/Build_all_with_remote_cache/stdout

What did you see happen?

signal 27 received on thread with no signal stack
fatal error: non-Go code disabled sigaltstack

runtime stack:
runtime.throw({0x10304c6af?, 0x0?})
	runtime/panic.go:1094 +0x34 fp=0x42ec56860 sp=0x42ec56830 pc=0x10219ebe4
runtime.noSignalStack(...)
	runtime/signal_unix.go:1106
runtime.adjustSignalStack2(0x1b, 0x42ec56918, 0x140797ed008, 0x1)
	runtime/signal_unix.go:599 +0xa0 fp=0x42ec56890 sp=0x42ec56860 pc=0x102180500
runtime.adjustSignalStack(0x1b, 0x140797ed008, 0x42ec56940)
	runtime/signal_unix.go:588 +0x230 fp=0x42ec56910 sp=0x42ec56890 pc=0x102180450
runtime.sigtrampgo(0x1b, 0x42ec56ab0, 0x42ec56b18)
	runtime/signal_unix.go:480 +0x84 fp=0x42ec56990 sp=0x42ec56910 pc=0x10217ff94
runtime.sigtrampgo(0x1b, 0x42ec56ab0, 0x42ec56b18)
	<autogenerated>:1 +0x1c fp=0x42ec569c0 sp=0x42ec56990 pc=0x1021a9d0c
runtime.sigtramp()
	runtime/sys_darwin_arm64.s:227 +0x4c fp=0x42ec56a80 sp=0x42ec569c0 pc=0x1021a884c

goroutine 17 gp=0x14000002700 m=1 mp=0x140000a8008 [syscall, locked to thread]:
runtime.goexit({})
	runtime/asm_arm64.s:1268 +0x4 fp=0x140000b6fe0 sp=0x140000b6fe0 pc=0x1021a7824

What did you expect to see?

no panic

[gabyhelp]

Related Issues

• runtime: signal 28 received on thread with no signal stack #22333 (closed)
• runtime: suspicious crash on `darwin/arm64`: "signal 16 received but handler not on signal stack" #75887 (closed)
• runtime: fatal error: semasleep on Darwin signal stack #75051
• runtime: stack split at bad time when disable inlining #20510 (closed)
• runtime: unexpected return pc for runtime.sigtramp on darwin/arm64 builder #29305 (closed)
• runtime: TestCgoCCodeSIGPROF fails intermittently on darwin/amd64 #16453 (closed)
• runtime: signal handling: sigtrampgo stack check fails in deferred TSAN handler call #18255 (closed)
• runtime,syscall: SIGABRT failures in syscall.StartProcess on darwin-arm64-12 starting 2022-09-03 #55072 (closed)
• runtime: "fatal error: signal_recv: inconsistent state" crash on ios/arm64 #46466 (closed)

Related Code Changes

• runtime: setg after sigFetchG

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[JunyangShao]

@golang/runtime

[ianlancetaylor]

The runtime error message is trying to tell you the problem: "fatal error: non-Go code disabled sigaltstack".

Quoting https://pkg.go.dev/os/signal:

If the non-Go code installs any signal handlers, it must use the SA_ONSTACK flag with sigaction. Failing to do so is likely to cause the program to crash if the signal is received. Go programs routinely run with a limited stack, and therefore set up an alternate signal stack.

In this case the signal is SIGPROF. Look for non-Go code linked into your program that is setting up a signal handler for SIGPROF.

[ukai]

No non-Go code linked into the program, as build CGO_ENABLED=0.

I think SIGPROF is used by runtime/pprof

[ianlancetaylor]

Yes, runtime/pprof uses SIGPROF.

If there is no non-Go code involved in this program, then this should be impossible. The Go runtime always installs signal handlers with SA_ONSTACK set. Is this problem reproducible? Is there a way that we can reproduce it ourselves?

[ukai]

It happened several times.

https://ci.chromium.org/ui/p/chromium/builders/build/ios-build-perf-developer/7889/overview
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8691654715001248849/+/u/Clean_builds/Build_chrome_without_remote_cache_with_Siso_in_native_mode/stdout

https://ci.chromium.org/ui/p/chromium/builders/build/ios-build-perf-siso/25663/overview
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8690040953658448481/+/u/Clean_builds/Build_all_without_remote_cache_with_Siso_disabling_clang_modules/stdout

https://ci.chromium.org/ui/p/chromium/builders/build/ios-build-perf-siso/25669/overview
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8690003051682517681/+/u/Clean_builds/Build_all_with_remote_cache/stdout

but no idea how to reproduce this yet.
happened rarely when pprof.StartCPUProfile is called?

$ go mod vendor
...
$ grep -r -i sigaltstack .
./vendor/golang.org/x/sys/unix/zsysnum_openbsd_386.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_mipsle.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_riscv64.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_mips64.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_amd64.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_mips64le.go
./vendor/golang.org/x/sys/unix/zsysnum_openbsd_riscv64.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_mips.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_arm.go
./vendor/golang.org/x/sys/unix/zsysnum_freebsd_386.go
./vendor/golang.org/x/sys/unix/zsysnum_zos_s390x.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_sparc64.go
./vendor/golang.org/x/sys/unix/zsysnum_freebsd_arm64.go
./vendor/golang.org/x/sys/unix/zsysnum_openbsd_ppc64.go
./vendor/golang.org/x/sys/unix/zsysnum_darwin_arm64.go
./vendor/golang.org/x/sys/unix/zsysnum_openbsd_arm64.go
./vendor/golang.org/x/sys/unix/zsysnum_openbsd_mips64.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_ppc64.go
./vendor/golang.org/x/sys/unix/zsysnum_dragonfly_amd64.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_s390x.go
./vendor/golang.org/x/sys/unix/zsysnum_freebsd_arm.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_arm64.go
./vendor/golang.org/x/sys/unix/zsysnum_freebsd_amd64.go
./vendor/golang.org/x/sys/unix/zsysnum_darwin_amd64.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_386.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_ppc64le.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_ppc.go
./vendor/golang.org/x/sys/unix/zsysnum_openbsd_arm.go
./vendor/golang.org/x/sys/unix/zsysnum_linux_loong64.go
./vendor/golang.org/x/sys/unix/zsysnum_freebsd_riscv64.go
./vendor/golang.org/x/sys/unix/zsysnum_openbsd_amd64.go
./vendor/github.com/ebitengine/purego/internal/fakecgo/symbols_linux.go
./vendor/github.com/ebitengine/purego/internal/fakecgo/go_netbsd.go
./vendor/github.com/ebitengine/purego/internal/fakecgo/symbols_darwin.go
./vendor/github.com/ebitengine/purego/internal/fakecgo/trampolines_stubs.s
./vendor/github.com/ebitengine/purego/internal/fakecgo/symbols_freebsd.go
./vendor/github.com/ebitengine/purego/internal/fakecgo/symbols.go
./vendor/github.com/ebitengine/purego/internal/fakecgo/symbols_netbsd.go

suspect github.com/ebitengine/purego ?

$ go mod why github.com/ebitengine/purego
# github.com/ebitengine/purego
go.chromium.org/build/siso/subcmd/collector
go.opentelemetry.io/collector/otelcol
go.opentelemetry.io/collector/service
go.opentelemetry.io/collector/service/internal/proctelemetry
github.com/shirou/gopsutil/v4/process
github.com/shirou/gopsutil/v4/internal/common
github.com/ebitengine/purego

[prattmic]

The purpose of purego is to call non-Go code (in an unsafe manner unsupported by the Go toolchain), so the libraries used from there would be my top suspects.

[cherrymui]

For now we suspect the issue is the C library that this program uses clears the SA_ONSTACK. Could you check if that is the case? Thanks.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)