runtime: cgo stuck because of go signal handler went into dead loop if pid=1

[cocktail828]

What version of Go are you using (go version)?

$ go version
go version go1.19 linux/amd64

Does this issue reproduce with the latest release?

Sure

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=
GONOSUMDB=
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2188387714=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The bug found in go1.10.3 at first, the I update go to go1.19, it happends too. I find the root case lives here:

go/src/runtime/signal_unix.go

Line 964 in de475e8

In almost all real cases the program is about to crash, but in my situation, it does not after usleep. But why, I cannot explain!!!.

Then I try fix it myself with the following patch:

	raise(sig)
	// Give the signal a chance to be delivered.
	// In almost all real cases the program is about to crash,
	// so sleeping here is not a waste of time.
	usleep(1000)

	switch sig {
	case _SIGSEGV, _SIGABRT, _SIGFPE, _SIGILL, _SIGBUS:
		print("fatal error: unexpected signal during runtime execution\n")
		signame := signame(sig)
		if signame != "" {
			print("[signal ", signame)
		} else {
			print("[signal ", hex(sig))
		}
		print(" code=", hex(c.sigcode()), " addr=", hex(c.sigaddr()), "]\n")
		dieFromSignal(sig)

	default:
		// If the signal didn't cause the program to exit, restore the
		// Go signal handler and carry on.
		//
		// We may receive another instance of the signal before we
		// restore the Go handler, but that is not so bad: we know
		// that the Go program has been ignoring the signal.
		setsig(sig, abi.FuncPCABIInternal(sighandler))
	}

What did you expect to see?

panic

What did you see instead?

See #56649

[seankhliao]

We will still need a way to reproduce this issue to verify any fixes

[prattmic]

What does “containered CentOS 7” mean? Does this change the semantics of rt_sigaction? We set the action for SIGSEGV to DFL, so it shouldn’t be possible to deliver to a handler anymore.

[cocktail828]

First thanks for your help.

What does “containered CentOS 7” mean? Does this change the semantics of rt_sigaction?

The container is built from centos 7.
The host is running kernel 3.10.0 and we don't make any change so rt_sigaction should work as expect.

We set the action for SIGSEGV to DFL, so it shouldn’t be possible to deliver to a handler anymore.

Yes, I can see that from strace. It should crash after usleep, but it doesn't. That's what I confused. Maybe the kernel bug???
The issue can only reproduce in small probility.

I can keep an environment for debug the next time it reproduce, if anybody interest enough to debug the issue.

[cocktail828]

I don't want to merge the patch, but want somebody professional enough to review the changes.

[ianlancetaylor]

Can you show us the output from strace -f? Thanks.

[prattmic]

@ianlancetaylor the strace is on #56649 (we probably should have reopened that one rather than making a new bug)

[ianlancetaylor]

Thanks. I also don't understand that strace. It appears to be running the code in raisebadsignal. That means that the SIGSEGV was received on a thread that was not started by Go code.

// unblocksig(sig)
[pid 108096] rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
// setsig(sig, handler) where handler == SIG_DFL
[pid 108096] rt_sigaction(SIGSEGV, {SIG_DFL, ~[], SA_RESTORER|SA_STACK|SA_RESTART|SA_SIGINFO, 0x7f695ef1d630}, NULL, 8) = 0
// raise(sig)
[pid 108096] getpid()                   = 1
[pid 108096] gettid()                   = 223
[pid 108096] tgkill(1, 223, SIGSEGV)    = 0
[pid 108096] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_TKILL, si_pid=1, si_uid=0} ---
// usleep(1000)
[pid 108096] nanosleep({0, 1000000}, NULL) = 0

In other words the process receives a SIGSEGV signal while the signal handler is set to SIG_DFL and the signal is not blocked. But the program just keeps running. That should not happen.

You said that this only happens occasionally. What does the strace -f output look like when the program does not go into an endless loop? Thanks.

[cocktail828]

In other words the process receives a SIGSEGV signal while the signal handler is set to SIG_DFL and the signal is not blocked. But the program just keeps running. That should not happen.

Yep. I agree with u. So this maybe the kernel bug, I guess. Recently, we also request our SRE to record the kernel version of the host which reproduce the issue.

You said that this only happens occasionally. What does the strace -f output look like when the program does not go into an endless loop? Thanks.

It take some times. I will provide the log and keep an environment for debug the next time it reproduce.
Thank you very much.

[cocktail828]

@ianlancetaylor
The following attachment is the output by strace -f -p ${pid}. May you get something useful from it.
strace-f.txt

Thanks in advance.

[ianlancetaylor]

Thanks. I just noticed si_pid=1 in the strace. I think that must be the problem. It sounds like you are running this program in a container, and you are running it as process 1. As it happens, process 1 is magic in Unix kernels: the kernel only delivers signals that actually have signal handlers. So when the Go runtime removes the signal handlers and raises the signal again, nothing happens. Now I don't know why this ever works at all, at least when running as process 1.

I don't know how much we care about this edge case. But we could change raisebadsignal to check for a process ID of 1 and a signal expected to kill the process.