Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: cgo stuck because of go signal handler went into dead loop #56649

Open
cocktail828 opened this issue Nov 8, 2022 · 7 comments
Open
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@cocktail828
Copy link

cocktail828 commented Nov 8, 2022

What version of Go are you using (go version)?

$ go version
go1.10.3

Does this issue reproduce with the latest release?

Sorry, I can not reproduce locally myself. But it occurs frequently on product enviroment.

What operating system and processor architecture are you using (go env)?

containered CentOS 7

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build951549230=/tmp/go-build -gno-record-gcc-switches"

The following is what I see.
Backtrace with gdb:

Thread 91 (Thread 0x7f64777fe700 (LWP 365)):
#0  runtime.usleep () at /usr/local/go/src/runtime/sys_linux_amd64.s:144
#1  0x000000000044435e in runtime.raisebadsignal (c=0x7f64777f5198, sig=11) at /usr/local/go/src/runtime/signal_unix.go:496
#2  0x0000000000444773 in runtime.badsignal (c=0x7f64777f5198, sig=11) at /usr/local/go/src/runtime/signal_unix.go:600
#3  0x0000000000443f58 in runtime.sigtrampgo (ctx=0x7f64777f5240, info=0x7f64777f5370, sig=11) at /usr/local/go/src/runtime/signal_unix.go:297
#4  0x000000000045efc3 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
#5  <signal handler called>
#6  0x00007f722fbba738 in __memcpy_ssse3_back () from /lib64/libc.so.6
#7  0x00007f705036f728 in AqcPropCalcModRun () from /ise-cn/bin/libaqc20.so
#8  0x00007f7050388092 in ModulesProcessBase () from /ise-cn/bin/libaqc20.so
#9  0x00007f705036baa7 in AqcInstAppend () from /ise-cn/bin/libaqc20.so
#10 0x00007f705036d4f4 in AQCAudioWrite () from /ise-cn/bin/libaqc20.so

Strace syscalls: As you can see, SIGSEGV is raised again and again...

[pid  7810] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7feb3bffe000} ---
[pid  7810] rt_sigprocmask(SIG_SETMASK, NULL, ~[KILL STOP], 8) = 0
[pid  7810] rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0
[pid  7810] sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
[pid  7810] sigaltstack({ss_sp=0xc4535d0000, ss_flags=0, ss_size=32768}, NULL) = 0
[pid  7810] rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE KILL SEGV TERM STKFLT CHLD STOP PROF SYS RTMIN RT_1], NULL, 8) = 0
[pid  7810] rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
[pid  7810] rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=~[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7ff8a5af3630}, NULL, 8) = 0
[pid  7810] tkill(178, SIGSEGV)         = 0
[pid  7810] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_TKILL, si_pid=1, si_uid=0} ---
[pid  7810] rt_sigaction(SIGSEGV, {sa_handler=0x45eff0, sa_mask=~[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7ff8a5af3630}, NULL, 8) = 0
[pid  7810] rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0
[pid  7810] sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}, NULL) = 0
[pid  7810] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP], NULL, 8) = 0
[pid  7810] rt_sigreturn({mask=[]})     = 140625886429224
[pid  7810] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7feb3bffe000} ---
[pid  7810] rt_sigprocmask(SIG_SETMASK, NULL, ~[KILL STOP], 8) = 0
[pid  7810] rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0
[pid  7810] sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
[pid  7810] sigaltstack({ss_sp=0xc4535d0000, ss_flags=0, ss_size=32768}, NULL) = 0
[pid  7810] rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE KILL SEGV TERM STKFLT CHLD STOP PROF SYS RTMIN RT_1], NULL, 8) = 0
[pid  7810] rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
[pid  7810] rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=~[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7ff8a5af3630}, NULL, 8) = 0
[pid  7810] tkill(178, SIGSEGV)         = 0
[pid  7810] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_TKILL, si_pid=1, si_uid=0} ---
[pid  7810] rt_sigaction(SIGSEGV, {sa_handler=0x45eff0, sa_mask=~[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7ff8a5af3630}, NULL, 8) = 0
[pid  7810] rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0
...

What did you expect to see?

Panic.

What did you see instead?

It stucked, no crash or panic.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 8, 2022
@cocktail828
Copy link
Author

cocktail828 commented Nov 8, 2022

@ianlancetaylor can you help me, thanks very much.

@tarikkilic
Copy link

tarikkilic commented Nov 8, 2022

We faced of the same issue. But when go test -msan command run, and error occured like this:

gcc: error: unrecognized argument to -fsanitize= option: 'memory'

It happens sporadically.

@cocktail828
Copy link
Author

cocktail828 commented Nov 8, 2022

@tarikkilic GCC does not support -fsanitize=memory. Try use clang instead.

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 8, 2022
@mknyszek mknyszek added this to the Backlog milestone Nov 8, 2022
@mknyszek
Copy link
Contributor

mknyszek commented Nov 8, 2022

Am I understanding correctly that the issue is occurring in Go 1.10.3? That version has not been supported for a very long time. Can you reproduce the issue with the latest release of Go?

CC @golang/runtime

@cocktail828
Copy link
Author

cocktail828 commented Nov 8, 2022

Yep. The issue occurs on product enviroment. We are trying recompile the docker images but it may take several days to update all services nodes and reproduce(if not fixed) the issue.
Thanks for your reply.

@cocktail828 cocktail828 changed the title runtime: cgo stucked because of go signal handler went into dead loop runtime: cgo stuck because of go signal handler went into dead loop Nov 8, 2022
@joedian
Copy link

joedian commented Nov 15, 2022

@cocktail828 can you confirm if the issue is still happening after recompiling?

@cocktail828
Copy link
Author

cocktail828 commented Nov 16, 2022

@cocktail828 can you confirm if the issue is still happening after recompiling?

Sorry for late response. It does not happened for now. I will keep trace the issue. If it does not hanppen for 2 weeks I will close the issue myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Status: No status
Development

No branches or pull requests

5 participants