Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: linux/amd64 crash in runtime.adjustSignalStack #56119

Closed
zyfxgo opened this issue Oct 10, 2022 · 5 comments
Closed

runtime: linux/amd64 crash in runtime.adjustSignalStack #56119

zyfxgo opened this issue Oct 10, 2022 · 5 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Backlog

Comments

@zyfxgo
Copy link

zyfxgo commented Oct 10, 2022 •
edited
Loading

What version of Go are you using (go version)?

# go version
go version go1.19.2 linux/amd64

Does this issue reproduce with the latest release?

yes, random crash, but high crash rate when high cpu load

What operating system and processor architecture are you using (go env)?

go env Output
# go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="//go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4153049028=/tmp/go-build -gno-record-gcc-switches"

What did you do?

use go1.19 and go1.19.2

What did you expect to see?

same problem

What did you see instead?

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/go_serving/main -env online_single_model -version GIT_611d8546_CI_138 -model_n'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000044fbca in runtime.adjustSignalStack (sig=23, mp=0xc002298400, gsigStack=0x7f4aafffc260, ~r0=<optimized out>) at /usr/local/go/src/runtime/signal_unix.go:546
546             if sp >= mp.gsignal.stack.lo && sp < mp.gsignal.stack.hi {
[Current thread is 1 (Thread 0x7f4aaffff700 (LWP 2571064))]
warning: File "/usr/local/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /usr/local/go/src/runtime/runtime-gdb.py
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
Installing openjdk unwinder
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 52, in <module>
    class Types(object):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 66, in Types
    nmethodp_t = gdb.lookup_type('nmethod').pointer()
gdb.error: No type named nmethod.
(gdb) bt
#0  0x000000000044fbca in runtime.adjustSignalStack (sig=23, mp=0xc002298400, gsigStack=0x7f4aafffc260, ~r0=<optimized out>) at /usr/local/go/src/runtime/signal_unix.go:546
#1  0x000000000044f99b in runtime.sigtrampgo (sig=23, info=0x7f4aafffc430, ctx=0x7f4aafffc300) at /usr/local/go/src/runtime/signal_unix.go:469
#2  0x000000000046fc46 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:359
#3  <signal handler called>
#4  pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ../sysdeps/unix/sysv/linux/pthread_sigmask.c:50
#5  0x00007f4af8f52dd7 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#6  0x00007f4af90d1df2 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#7  0x00007f4af90d2251 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#8  0x00007f4af8ce6273 in ?? () from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#9  0x00007f4af9630770 in hdfsThreadDestructor (v=0x7f4a2c2dd530) at /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/os/posix/thread_local_storage.c:68
#10 0x00007f4afa97a408 in __nptl_deallocate_tsd () at pthread_create.c:300
#11 0x00007f4afa97b81b in __nptl_deallocate_tsd () at ../sysdeps/nptl/futex-internal.h:200
#12 start_thread (arg=0x7f4aaffff700) at pthread_create.c:473
#13 0x00007f4afa6a461f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
-----
info all
rax            0x17     23
rbx            0xc002298400     824669996032
rcx            0x7f4aafffc260   139958757081696
rdx            0x0      0
rsi            0x7f4aafffc238   139958757081656
rdi            0x101    257
rbp            0x7f4aafffc228   0x7f4aafffc228
rsp            0x7f4aafffc1d0   0x7f4aafffc1d0
r8             0x1a19760        27367264
r9             0x1      1
r10            0x8      8
r11            0x246    582
r12            0x7f4a2c196490   139956544169104
r13            0x7f4af95dc440   139959987979328
r14            0x0      0
r15            0x7f4afab8e340   139960010728256
rip            0x44fbca 0x44fbca <runtime.adjustSignalStack+42>
eflags         0x10202  [ IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
st0            0        (raw 0x00000000000000000000)
st1            0        (raw 0x00000000000000000000)
st2            0        (raw 0x00000000000000000000)
st3            0        (raw 0x00000000000000000000)
st4            0        (raw 0x00000000000000000000)
st5            0        (raw 0x00000000000000000000)
st6            0        (raw 0x00000000000000000000)
st7            0        (raw 0x00000000000000000000)
fctrl          0x37f    895
fstat          0x0      0
ftag           0xffff   65535
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
mxcsr          0x1f80   [ IM DM ZM OM UM PM ]
k0             0x0      0
k1             0x0      0
k2             0x0      0
k3             0x0      0
k4             0x0      0
k5             0x0      0
k6             0x0      0
k7             0x0      0
pkru           0x55555554       1431655764
zmm0           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm1           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm2           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm3           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm4           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm5           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm6           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm7           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm8           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm9           {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm10          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm11          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm12          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm13          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm14          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm15          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm16          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm17          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm18          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm19          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm20          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm21          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm22          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm23          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm24          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm25          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm26          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm27          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm28          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm29          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm30          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
zmm31          {v16_float = {0x0 <repeats 16 times>}, v8_double = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v64_int8 = {0x0 <repeats 64 times>}, v32_int16 = {0x0 <repeats 32 times>}, v16_int32 = {
    0x0 <repeats 16 times>}, v8_int64 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int128 = {0x0, 0x0, 0x0, 0x0}}
-----
(gdb) p mp.gsignal
$2 = (runtime.g *) 0x0
-----
(gdb) p sp
$4 = 139958757081656
-----
(gdb) p *mp
$5 = {g0 = 0xc007faf380, morebuf = {sp = 0, pc = 0, g = 0, ctxt = 0x0, ret = 0, lr = 0, bp = 0}, divmod = 0, _ = 0, procid = 2571064, gsignal = 0x0, goSigStack = {stack = {lo = 0, hi = 0},
    stackguard0 = 0, stackguard1 = 0, stktopsp = 0}, sigmask = {0, 0}, tls = {0, 0, 0, 0, 0, 0}, mstartfn = {void (void)} 0xc002298400, curg = 0x0, caughtsig = 0, p = 0, nextp = 0, oldp = 0,
  id = 151945, mallocing = 0, throwing = 0, preemptoff = 0x0 "", locks = 0, dying = 0, profilehz = 0, spinning = false, blocked = false, newSigstack = true, printlock = 0 '\000', incgo = false,
  freeWait = 0, fastrand = 16604656581019358215, needextram = false, traceback = 0 '\000', ncgocall = 3, ncgo = 0, cgoCallersUse = 0, cgoCallers = 0xc007e5df00, park = {key = 0},
  alllink = 0xc003801400, schedlink = 824767582208, lockedg = 0, createstack = {0 <repeats 32 times>}, lockedExt = 1, lockedInt = 0, nextwaitm = 0, waitunlockf = {void (runtime.g *, void *,
    bool *)} 0xc002298400, waitlock = 0x0, waittraceev = 25 '\031', waittraceskip = 4, startingtrace = false, syscalltick = 1371398, freelink = 0x0, libcall = {fn = 0, n = 0, args = 0, r1 = 0,
    r2 = 0, err = 0}, libcallpc = 0, libcallsp = 0, libcallg = 0, syscall = {fn = 0, n = 0, args = 0, r1 = 0, r2 = 0, err = 0}, vdsoSP = 0, vdsoPC = 0, preemptGen = 17, signalPending = 1,
  dlogPerM = {<No data fields>}, mOS = {profileTimer = 0, profileTimerValid = 0, needPerThreadSyscall = {noCopy = {<No data fields>}, value = 0 '\000'}}, locksHeldLen = 0, locksHeld = {{lockAddr = 0,
      rank = 0}, {lockAddr = 0, rank = 0}, {lockAddr = 0, rank = 0}, {lockAddr = 0, rank = 0}, {lockAddr = 0, rank = 0}, {lockAddr = 0, rank = 0}, {lockAddr = 0, rank = 0}, {lockAddr = 0, rank = 0}, {
      lockAddr = 0, rank = 0}, {lockAddr = 0, rank = 0}}}
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Oct 10, 2022
@zyfxgo
Copy link
Author

zyfxgo commented Oct 10, 2022 

(gdb) disas
Dump of assembler code for function runtime.adjustSignalStack:
   0x000000000044fba0 <+0>:     sub    $0x60,%rsp
   0x000000000044fba4 <+4>:     mov    %rbp,0x58(%rsp)
   0x000000000044fba9 <+9>:     lea    0x58(%rsp),%rbp
   0x000000000044fbae <+14>:    mov    %rbx,0x70(%rsp)
   0x000000000044fbb3 <+19>:    mov    %rcx,0x78(%rsp)
   0x000000000044fbb8 <+24>:    mov    %eax,0x68(%rsp)
   0x000000000044fbbc <+28>:    mov    0x50(%rbx),%rdx
   0x000000000044fbc0 <+32>:    lea    0x68(%rsp),%rsi
   0x000000000044fbc5 <+37>:    mov    %rsi,0x10(%rsp)
=> 0x000000000044fbca <+42>:    cmp    %rsi,(%rdx)
   0x000000000044fbcd <+45>:    ja     0x44fbe1 <runtime.adjustSignalStack+65>
   0x000000000044fbcf <+47>:    cmp    %rsi,0x8(%rdx)
   0x000000000044fbd3 <+51>:    jbe    0x44fbe1 <runtime.adjustSignalStack+65>
   0x000000000044fbd5 <+53>:    xor    %eax,%eax
   0x000000000044fbd7 <+55>:    mov    0x58(%rsp),%rbp
   0x000000000044fbdc <+60>:    add    $0x60,%rsp
   0x000000000044fbe0 <+64>:    retq
   0x000000000044fbe1 <+65>:    movups %xmm15,0x40(%rsp)
   0x000000000044fbe7 <+71>:    movups %xmm15,0x48(%rsp)
   0x000000000044fbed <+77>:    movq   $0x0,(%rsp)
   0x000000000044fbf5 <+85>:    lea    0x40(%rsp),%rax
   0x000000000044fbfa <+90>:    mov    %rax,0x8(%rsp)
   0x000000000044fbff <+95>:    nop
   0x000000000044fc00 <+96>:    callq  0x46ffa0 <runtime.sigaltstack>
   0x000000000044fc05 <+101>:   xorps  %xmm15,%xmm15
   0x000000000044fc09 <+105>:   mov    %fs:0xfffffffffffffff8,%r14
   0x000000000044fc12 <+114>:   mov    0x40(%rsp),%rax
   0x000000000044fc17 <+119>:   mov    0x48(%rsp),%ecx
   0x000000000044fc1b <+123>:   bt     $0x1,%ecx
   0x000000000044fc1f <+127>:   nop
   0x000000000044fc20 <+128>:   jb     0x44fcf9 <runtime.adjustSignalStack+345>
   0x000000000044fc26 <+134>:   mov    0x10(%rsp),%rcx
   0x000000000044fc2b <+139>:   cmp    %rcx,%rax
   0x000000000044fc2e <+142>:   ja     0x44fcfe <runtime.adjustSignalStack+350>
   0x000000000044fc34 <+148>:   mov    0x50(%rsp),%rdx
   0x000000000044fc39 <+153>:   add    %rax,%rdx
   0x000000000044fc3c <+156>:   nopl   0x0(%rax)
   0x000000000044fc40 <+160>:   cmp    %rdx,%rcx
   0x000000000044fc43 <+163>:   jae    0x44fcfe <runtime.adjustSignalStack+350>
   0x000000000044fc49 <+169>:   nop
   0x000000000044fc4a <+170>:   mov    %r14,0x20(%rsp)
   0x000000000044fc4f <+175>:   mov    0x78(%rsp),%rcx
   0x000000000044fc54 <+180>:   test   %rcx,%rcx

@joedian joedian added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 10, 2022
@joedian joedian added this to the Backlog milestone Oct 10, 2022
@cherrymui
Copy link
Member

Could you share what your program does?

#2  0x000000000046fc46 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:359
#3  <signal handler called>
#4  pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ../sysdeps/unix/sysv/linux/pthread_sigmask.c:50

It looks like this is a non-Go created thread but has Go signal handler registered, and while it is setting the signal masks it gets signaled and the Go signal handler runs?

@cherrymui cherrymui added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 10, 2022
@zyfxgo
Copy link
Author

zyfxgo commented Oct 11, 2022

I am not sure because I use a private dynamic link library (libserving_processor.so).

module go_serving

go 1.19

require (
	github.com/apache/thrift v0.13.0
	github.com/colinmarc/hdfs/v2 v2.2.0
	github.com/gin-contrib/pprof v1.4.0
	github.com/gin-gonic/gin v1.8.1
	github.com/go-redis/redis/v8 v8.11.5
	github.com/golang/protobuf v1.5.2
	github.com/prometheus/common v0.26.0
	github.com/rs/cors/wrapper/gin v0.0.0-20220619195839-da52b0701de5
	github.com/samuel/go-zookeeper v0.0.0-20201211165307-7117e9ea2414
	github.com/sirupsen/logrus v1.9.0
	<private_deps_v1.4.3>
	google.golang.org/grpc v1.26.0
	google.golang.org/protobuf v1.28.1

)

require (
	github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect
	github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 // indirect
	github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d // indirect
	github.com/aliyun/alibaba-cloud-sdk-go v1.61.18 // indirect
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23 // indirect
	github.com/cespare/xxhash/v2 v2.1.2 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/gin-contrib/sse v0.1.0 // indirect
	github.com/go-errors/errors v1.0.1 // indirect
	github.com/go-ole/go-ole v1.2.6 // indirect
	github.com/go-playground/locales v0.14.0 // indirect
	github.com/go-playground/universal-translator v0.18.0 // indirect
	github.com/go-playground/validator/v10 v10.10.0 // indirect
	github.com/goccy/go-json v0.9.7 // indirect
	github.com/hashicorp/go-uuid v1.0.2 // indirect
	github.com/jcmturner/aescts/v2 v2.0.0 // indirect
	github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
	github.com/jcmturner/gofork v1.0.0 // indirect
	github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
	github.com/jcmturner/gokrb5/v8 v8.4.2 // indirect
	github.com/jcmturner/rpc/v2 v2.0.3 // indirect
	github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af // indirect
	github.com/json-iterator/go v1.1.12 // indirect
	github.com/leodido/go-urn v1.2.1 // indirect
	github.com/lestrrat/go-file-rotatelogs v0.0.0-20180223000712-d3151e2a480f // indirect
	github.com/lestrrat/go-strftime v0.0.0-20180220042222-ba3bf9c1d042 // indirect
	github.com/mattn/go-isatty v0.0.14 // indirect
	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/nacos-group/nacos-sdk-go v1.0.9 // indirect
	github.com/pelletier/go-toml/v2 v2.0.1 // indirect
	github.com/pkg/errors v0.9.1 // indirect
	github.com/prometheus/client_golang v1.9.0 // indirect
	github.com/prometheus/client_model v0.2.0 // indirect
	github.com/prometheus/procfs v0.2.0 // indirect
	github.com/robfig/cron/v3 v3.0.1 // indirect
	github.com/rs/cors v1.8.1 // indirect
	github.com/shirou/gopsutil/v3 v3.21.6 // indirect
	github.com/tklauser/go-sysconf v0.3.6 // indirect
	github.com/tklauser/numcpus v0.2.2 // indirect
	github.com/toolkits/concurrent v0.0.0-20150624120057-a4371d70e3e3 // indirect
	github.com/ugorji/go/codec v1.2.7 // indirect
	go.uber.org/atomic v1.6.0 // indirect
	go.uber.org/multierr v1.5.0 // indirect
	go.uber.org/zap v1.15.0 // indirect
	golang.org/x/crypto v0.0.0-20221005025214-4161e89ecf1b // indirect
	golang.org/x/net v0.0.0-20221004154528-8021a29435af // indirect
	golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 // indirect
	golang.org/x/text v0.3.7 // indirect
	google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 // indirect
	gopkg.in/alecthomas/kingpin.v2 v2.2.6 // indirect
	gopkg.in/ini.v1 v1.42.0 // indirect
	gopkg.in/yaml.v2 v2.4.0 // indirect
)

<private_deps_v1.4.3>
go 1.16

require (
	github.com/alibaba/sentinel-golang v1.0.3
	github.com/aliyun/aliyun-datahub-sdk-go v0.1.3
	github.com/apache/thrift v0.13.0
	github.com/go-redis/redis/v8 v8.11.4
	github.com/go-sql-driver/mysql v1.5.0
	github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6
	github.com/golang/protobuf v1.5.2
	github.com/hashicorp/go-multierror v1.1.1
	github.com/json-iterator/go v1.1.12
	github.com/lib/pq v1.0.0
	github.com/nacos-group/nacos-sdk-go v1.0.9
	github.com/onsi/ginkgo v1.16.5 // indirect
	github.com/onsi/gomega v1.17.0 // indirect
	github.com/pkg/errors v0.9.1
	github.com/prometheus/client_golang v1.9.0
	github.com/prometheus/common v0.26.0
	github.com/robfig/cron/v3 v3.0.1
	github.com/samuel/go-zookeeper v0.0.0-20201211165307-7117e9ea2414
	github.com/shirou/gopsutil v3.21.11+incompatible
	github.com/shirou/gopsutil/v3 v3.21.6
	github.com/shopspring/decimal v1.3.1 // indirect
	github.com/sirupsen/logrus v1.6.0
	github.com/smartystreets/assertions v1.2.1
	github.com/stretchr/testify v1.7.1
	github.com/yusufpapurcu/wmi v1.2.2 // indirect
	go.uber.org/atomic v1.6.0
	golang.org/x/net v0.0.0-20210428140749-89ef3d95e781
	golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40 // indirect
	google.golang.org/grpc v1.26.0
	google.golang.org/protobuf v1.28.1 // indirect
)

how can i find out (where Go signal handler registered)?
why crash? does it meet expectations?

@prattmic
Copy link
Member

A couple of questions:

  1. Do these crashes always happen under a __nptl_deallocate_tsd frame? (i.e., the thread is exiting)
  2. Have you ever seen a "non-Go code set up signal handler without SA_ONSTACK flag" crash? Go requires signal handlers have SA_ONSTACK set, but the JVM doesn't always set this consistently. Go tries to fix this up, but can't if initialization happens in the wrong order.

@zyfxgo
Copy link
Author

zyfxgo commented Oct 13, 2022

yes i get it.
I create a thread in c to call c function instead, It works.
no crash found since two days ago, it may be solved?

@zyfxgo zyfxgo closed this as completed Oct 21, 2022
@zyfxgo zyfxgo moved this to Done in Go Compiler / Runtime Oct 21, 2022
@golang golang locked and limited conversation to collaborators Oct 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@prattmic @gopherbot @zyfxgo @cherrymui @joedian