runtime: in core dump generated for a signal while executing C code, gdb reports a corrupt stack

[MariappanBalraj]

What version of Go are you using (go version)?

$ go version
go version go1.19.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1839558467=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Following is the working source code of GO which uses CGO. From GO code, the c function test3() is called, which calls test2(), which calls test1(). In test1(), there is a NULL pointer assignment, which will cause segmentation fault. To get the core dump, "ulimit -c unlimited" and "echo "core" > /proc/sys/kernel/core_pattern" are run. The code is compiled without optimization by setting env variable CGO_CFLAGS to -g. The program is run by "GOTRACEBACK=crash ./test", which produced core dump. From the core dump, I am not getting complete C stack and it reports corrupt stack. Please note that when I run the program by using gdb, I am getting the complete stack.

package main

/*
#include <stdio.h>

void test1(void) {
int p = (int)NULL;
*p = 30;
}

void test2(void) {
int val = 2;
test1();
}

void test3(void) {
int val = 3;
test2();
}
*/
import "C"

func main() {
C.test3()
}

What did you expect to see?

I am expecting the same C stack which is displayed when the program is run directly by gdb.

gdb ./test
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test...
warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/ubuntu/mbalraj/GO/TEST/test.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) run
Starting program: /home/ubuntu/mbalraj/GO/TEST/test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffd08f7640 (LWP 193717)]
[New Thread 0x7fffcbfff640 (LWP 193718)]
[New Thread 0x7fffcb7fe640 (LWP 193719)]
[New Thread 0x7fffcaffd640 (LWP 193720)]
[New Thread 0x7fffca7fc640 (LWP 193721)]

Thread 1 "test" received signal SIGSEGV, Segmentation fault.
0x000000000045abe6 in test1 () at /home/ubuntu/GO/TEST/test.go:8
8 *p = 30;
(gdb) bt
#0 0x000000000045abe6 in test1 () at /home/ubuntu/GO/TEST/test.go:8
#1 0x000000000045ac07 in test2 () at /home/ubuntu/GO/TEST/test.go:13
#2 0x000000000045ac22 in test3 () at /home/ubuntu/GO/TEST/test.go:18
#3 0x000000000045ac42 in _cgo_6ac9acc88c92_Cfunc_test3 (v=0xc000056f70) at /tmp/go-build/cgo-gcc-prolog:49
#4 0x00000000004567e4 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:844
#5 0x00000000004e15e0 in ?? ()
#6 0x0000000000000001 in ?? ()
#7 0x000000c000100a00 in ?? ()
#8 0x00007fffffffe2d8 in ?? ()
#9 0x0000000000439fa5 in runtime.malg.func1 () at /usr/local/go/src/runtime/proc.go:4080
#10 0x0000000000456629 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:492
#11 0x0000000000458fe5 in runtime.newproc (fn=0x1) at :1
#12 0x00000000004c9780 in runtime[scavenger] ()
#13 0x0000000000000001 in ?? ()
#14 0x0000000000456525 in runtime.mstart () at /usr/local/go/src/runtime/asm_amd64.s:390
#15 0x00000000004564af in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:354
#16 0x0000000000000001 in ?? ()
#17 0x00007fffffffe458 in ?? ()
#18 0x0000000000000000 in ?? ()

What did you see instead?

gdb ./test core.193685
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test...
[New LWP 193685]
[New LWP 193687]
[New LWP 193686]
[New LWP 193689]
[New LWP 193688]
[New LWP 193690]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by ./test'. Program terminated with signal SIGABRT, Aborted. #0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159 159 RET [Current thread is 1 (Thread 0x7fdf0cfb7740 (LWP 193685))] warning: Unsupported auto-load script at offset 0 in section .debug_gdb_scripts of file /home/ubuntu/mbalraj/GO/TEST/test. Use info auto-load python-scripts [REGEXP]' to list them.
(gdb) bt
#0 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
#1 0x0000000000443345 in runtime.dieFromSignal (sig=6) at /usr/local/go/src/runtime/signal_unix.go:870
#2 0x00000000004438de in runtime.sigfwdgo (sig=6, info=, ctx=, ~r0=)
at /usr/local/go/src/runtime/signal_unix.go:1086
#3 0x0000000000442027 in runtime.sigtrampgo (sig=0, info=0x0, ctx=0x4583c1 <runtime.raise+33>) at /usr/local/go/src/runtime/signal_unix.go:432
#4 0x00000000004586a6 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:359
#5
#6 runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
#7 0x0000000000443345 in runtime.dieFromSignal (sig=6) at /usr/local/go/src/runtime/signal_unix.go:870
#8 0x0000000000443558 in runtime.crash () at /usr/local/go/src/runtime/signal_unix.go:962
#9 0x000000000042f891 in runtime.fatalthrow.func1 () at /usr/local/go/src/runtime/panic.go:1129
#10 0x000000000042f80c in runtime.fatalthrow (t=) at /usr/local/go/src/runtime/panic.go:1122
#11 0x000000000042f4bd in runtime.throw (s=...) at /usr/local/go/src/runtime/panic.go:1047
#12 0x0000000000443289 in runtime.sigpanic () at /usr/local/go/src/runtime/signal_unix.go:819
#13 0x000000000045abe6 in test1 () at /home/ubuntu/GO/TEST/test.go:8
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[ianlancetaylor]

CC @golang/runtime

Looks like gdb can't unwind past the call to sigpanic. It gets to test1 but can't get any farther. This might be difficult to solve, as the call to sigpanic is dynamically generated.

[MariappanBalraj]

Hi Ian, When I simulated the crash by adding assert, from core dump, I am getting the complete stack. package main /* #include <stdio.h> #include <assert.h> void test1(void) { assert(1 == 2); } void test2(void) { int val = 2; test1(); } void test3(void) { int val = 3; test2(); } */ import "C" func main() { C.test3() } (gdb) bt #0 runtime.usleep () at /usr/local/go/src/runtime/sys_linux_amd64.s:140 #1 0x0000000000442e2f in runtime.sighandler (sig=<optimized out>, info=<optimized out>, ctxt=<optimized out>, gp=0xc000006340) at /usr/local/go/src/runtime/signal_unix.go:789 #2 0x00000000004421d4 in runtime.sigtrampgo (sig=6, info=0xc00000fbf0, ctx=0xc00000fac0) at /usr/local/go/src/runtime/signal_unix.go:479 #3 0x00000000004586c6 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:359 #4 <signal handler called> #5 0x00007f78b7524a7c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x00007f78b74d0476 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x00007f78b74b67f3 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #8 0x00007f78b74b671b in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x00007f78b74c7e96 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6 #10 0x000000000045ac22 in test1 () at /home/ubuntu/mbalraj/GO/TEST/TEST3/test.go:8 #11 0x000000000045ac3a in test2 () at /home/ubuntu/mbalraj/GO/TEST/TEST3/test.go:13 #12 0x000000000045ac55 in test3 () at /home/ubuntu/mbalraj/GO/TEST/TEST3/test.go:18 #13 0x000000000045ac75 in _cgo_601fe3e3f98a_Cfunc_test3 (v=0xc00005bf70) at /tmp/go-build/cgo-gcc-prolog:49 #14 0x0000000000456804 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:844 #15 0x00000000004e15e0 in ?? () #16 0x0000000000000001 in ?? () #17 0x000000c000040a00 in ?? () #18 0x00007ffe126b9c08 in ?? () #19 0x0000000000439fc5 in runtime.malg.func1 () at /usr/local/go/src/runtime/proc.go:4080 #20 0x0000000000456649 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:492 #21 0x0000000000459005 in runtime.newproc (fn=0x1) at <autogenerated>:1 #22 0x00000000004c9780 in runtime[scavenger] () #23 0x0000000000000001 in ?? () #24 0x0000000000456545 in runtime.mstart () at /usr/local/go/src/runtime/asm_amd64.s:390 #25 0x00000000004564cf in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:354 #26 0x0000000000000001 in ?? () #27 0x00007ffe126b9d88 in ?? () #28 0x0000000000000000 in ?? () Best Regards Mariappan

…

On Tue, Jan 10, 2023 at 3:31 AM Ian Lance Taylor ***@***.***> wrote: CC @golang/runtime <https://github.com/orgs/golang/teams/runtime> Looks like gdb can't unwind past the call to sigpanic. It gets to test1 but can't get any farther. This might be difficult to solve, as the call to sigpanic is dynamically generated. — Reply to this email directly, view it on GitHub <#57698 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A25ROZXH57VPVZHRMPDIBZ3WRSDDTANCNFSM6AAAAAATVHEFR4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ianlancetaylor]

I'm suggesting that the problem is that gdb can't unwind past a call to sigpanic. When you call the C function assert, no call to sigpanic is involved.

[MariappanBalraj]

Hi Ian, Is there any way to bypass sigpanic? Best Regards Mariappan

…

On Wed, 11 Jan, 2023, 12:03 am Ian Lance Taylor, ***@***.***> wrote: I'm suggesting that the problem is that gdb can't unwind past a call to sigpanic. When you call the C function assert, no call to sigpanic is involved. — Reply to this email directly, view it on GitHub <#57698 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A25ROZXHHIGE5E4X2NXGGV3WRWTPNANCNFSM6AAAAAATVHEFR4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cherrymui]

Maybe we could consider not injecting a sigpanic if a faulting signal lands on a non-user G. It would be that either we're in the runtime, or we're running some C code. Either way I think we cannot recover.

[MariappanBalraj]

My understanding is when there is a segmentation fault in the C code, sigpanic handler is GO code. So, we have GO stack and C stack and signal handler stack which is another GO stack. I am trying to understand what is done by injecting sigpanic. What kind of recovery is done? I am able to think about handling defer calls till the main function. In C, there is no such kind of mechanism available. But we are ok with that. I am able to understand that some people may need it and it is also one of the best feature of GO. But, when we start using CGO, if there is any issue then we should be able to debug it. Debugging support is very very important. Otherwise, the end customer will not be happy about it. Is there any API currently available to bypass injecting sigpanic? If there is no such API then could you please provide an API? We can call the API in the beginning of the program. Or can it be a build option? Best Regards Mariappan

…

On Wed, Jan 11, 2023 at 9:21 PM cherrymui ***@***.***> wrote: Maybe we could consider not injecting a sigpanic if a faulting signal lands on a non-user G. It would be that either we're in the runtime, or we're running some C code. Either way I think we cannot recover. — Reply to this email directly, view it on GitHub <#57698 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A25ROZXJMC6DUQNDTY3OHE3WR3JITANCNFSM6AAAAAATVHEFR4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ianlancetaylor]

There is no API to avoid calling sigpanic, and we are not going to add one.

@cherrymui has pointed out that it is currently impossible to recover a panic that occurs while executing C code, so it may be reasonable to avoid injecting a sigpanic call in C code. We could, instead, simply reraise the signal, as we do for other signals that occur while executing C code. See the badsignal function.

However, making that change is not going to be a high priority for us. I don't know of anybody else who has reported problems getting a stack backtrace in a core dump generated while executing C code called by Go code. Go is an open source project, and anybody is welcome to contribute. See https://go.dev/doc/contribute.