runtime: crash after shrinkstack crash with cgo

[codecor-cn]

Go version

go version go1.26.4 (Red Hat 1.26.4-1.el10_2) linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE='on'
GOAMD64='v3'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/shiyu/.cache/go-build'
GOCACHEPROG=''
GODEBUG='gcshrinkstackoff=1'
GOENV='/home/shiyu/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1711678839=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/shiyu/gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/shiyu/gopath'
GOPRIVATE=''
GOPROXY='http://goproxy.cn,http://goproxy.io,direct'
GOROOT='/usr/lib/golang'
GOSUMDB='off'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/shiyu/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib/golang/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.26.4 (Red Hat 1.26.4-1.el10_2)'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

We build a Go application which also include some CGO requests and very frequently we see errors produced by an unwinding crash. Unfortunately, we are not able to reproduce the crash but the crashes occurred often without executing any specific request to produce this error.
we use cgo call c++ lib repo is https://github.com/pseudocodes/go2ctp at 6.7.13 branch
go mod is 1.25.9

What did you see happen?

The program experiences two types of crashes. AI analysis reveals that both crashes stem from the same root cause: after stack contraction, the stack space fails to replicate smoothly, leading to incorrect handling of certain stack frames. This is evident from the fact that the stack sizes in both crash logs have become 4kb, whereas the initial stack size for Go language goroutines is 8kb. Furthermore, after utilizing the GODEBUG=gcshrinkstackoff=1 environment variable, the program runing not crash

The first crash log:

0xc6a2b881000 - 0xc6a2b880000 = 0x1000 = 4KB

runtime: g 10: unexpected return pc for xxxxxxxxxxxxxx called from 0xc6a2bbbbb20
stack: frame={sp:0xc6a2b880db0, fp:0xc6a2b880fc8} stack=[0xc6a2b880000,0xc6a2b881000)
fatal error: unknown caller pc

goroutine 10 gp=0xc6a2b8b4960 m=10 mp=0xc6a2ce80008 [running]:
runtime.systemstack_switch()
	/usr/lib/golang/src/runtime/asm_amd64.s:516 +0x8 fp=0xc6a2b880c40 sp=0xc6a2b880c30 pc=0x48ef28
runtime.(*_panic).nextFrame(0x47060b?)
	/usr/lib/golang/src/runtime/panic.go:1003 +0x58 fp=0xc6a2b880c80 sp=0xc6a2b880c40 pc=0x450318
runtime.(*_panic).start(0x8?, 0x0?, 0x4170e4?)
	/usr/lib/golang/src/runtime/panic.go:924 +0x145 fp=0xc6a2b880ca0 sp=0xc6a2b880c80 pc=0x450125
panic({0xa3e600?, 0x11026d0?})
	/usr/lib/golang/src/runtime/panic.go:854 +0x130 fp=0xc6a2b880d50 sp=0xc6a2b880ca0 pc=0x488f70
runtime.panicmem(...)
	/usr/lib/golang/src/runtime/panic.go:336
runtime.sigpanic()
	/usr/lib/golang/src/runtime/signal_unix.go:931 +0x378 fp=0xc6a2b880db0 sp=0xc6a2b880d50 pc=0x48b5d8
xxxxxxxxxxxxxx(0xc6a2b881db0)
	/home/shiyu/code/yyyyyy:203 +0x245 fp=0xc6a2b880fc8 sp=0xc6a2b880db0 pc=0x586d45
runtime: g 10: unexpected return pc for xxxxxxxxxxxxxx called from 0xc6a2bbbbb20
stack: frame={sp:0xc6a2b880db0, fp:0xc6a2b880fc8} stack=[0xc6a2b880000,0xc6a2b881000)

The second crash log:

0x2f2fc170000 - 0x2f2fc16f000 = 0x1000 = 4KB

runtime: g56: frame.sp=0x2f2fc16feb0 top=0x2f2fc16ffe0
	stack=[0x2f2fc16f000-0x2f2fc170000
fatal error: traceback did not unwind completely

goroutine 56 gp=0x2f2efff2b40 m=nil [chan receive (scan), 72 minutes]:
runtime.gopark(0x0?, 0x2f2ed400220?, 0xf?, 0x0?, 0x2f2fb890c96?)
	/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x2f2fc16fe10 sp=0x2f2fc16fdf0 pc=0x4894ce
runtime.chanrecv(0x2f2ed498380, 0x2f2fc16ffb0, 0x1)
	/usr/lib/golang/src/runtime/chan.go:667 +0x4ae fp=0x2f2fc16fe88 sp=0x2f2fc16fe10 pc=0x41e2ae
runtime.chanrecv2(0xb47c96?, 0x6?)
	/usr/lib/golang/src/runtime/chan.go:514 +0x12 fp=0x2f2fc16feb0 sp=0x2f2fc16fe88 pc=0x41ddf2
oooooooooooooooooooooooooooooooooooooooo(0x0)
	/home/shiyu/code/iiiiiiiiiiiiiiiiiiiiiiiii:275 +0x96 fp=0x2f2fc16ffc8 sp=0x2f2fc16feb0 pc=0x587396
created by pppppppppppppppppppp in goroutine 1
	/home/shiyu/code/jjjjjjjjjjjjjjjjjjjjjjjjj:69 +0x4c5

What did you expect to see?

Can this crash be confirmed as an bug?
After the fix, can I operate normally without using GODEBUG=gcshrinkstackoff=1 environment variables

[gabyhelp]

Related Issues

• runtime: spurious unknown caller pc in cgo application #59538 (closed)
• runtime: panic not generating a correct backtrace stack while crashing in cgo #63277 (closed)
• runtime: unexpected return pc for runtime.gcBgMarkWorker called from 0x0 #57550 (closed)
• runtime: systemstack called from unexpected goroutine/traceback has leftover defers #31538 (closed)
• runtime: panic not generating a backtrace stack correctly while crashing in cgo coredump - Go 1.24 #76634 (closed)
• runtime: unexpected return pc crash on linux-amd64-alpine builder #54306 (closed)
• runtime: unexpected return pc for runtime.sigpanic called from 0x0 #36287 (closed)
• runtime/traceback: segmentation violation failures from unwinding crash #64030
• misc/cgo/test: linux-amd64-alpine: "fatal error: unknown caller pc" in gentraceback #54749 (closed)
• runtime: unable to backtrace program using Go shared object #52608 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

My guess would be that you are violating cgo safety rules, resulting in the C side holding a reference into a Go stack after that stack moves. See https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers for full details.

[codecor-cn]

My guess would be that you are violating cgo safety rules, resulting in the C side holding a reference into a Go stack after that stack moves. See https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers for full details.

I tried to investigate the direction of the problem you raised, using parameters such as "- asan" "GODEBUG=cgocheck=2 and invalidptr=1", but there was no result. After using the "GODEBUG=gcshrinkstackoff=1" parameter, the problem disappeared, so I suspect that there may be some conflict between shrinkstack and CGO

[ianlancetaylor]

Nobody else is reporting such a problem.

Are you using the #cgo noescape or #cgo nocallback directives?

Is there a way that we can reproduce the problem ourselves?

[codecor-cn]

I use cgo through this repo https://github.com/pseudocodes/go2ctp@6.7.13, did not use the #cgo noescape #cgo nocallback.
There are similar prompts for this question #64030
and need me to release the execution code and investigate the cause of the crash? If necessary, I need about half a month to delete private code

[ianlancetaylor]

Thanks for the code reference. Unfortunately that is a complex piece of code. Perhaps somebody will see a problem based on the stack traceback and the stack shrinking code. It would certainly be much simpler to find the problem if there is a way that we can reproduce it.

CC @golang/runtime