Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: unexpected return pc for golang.gopark #47003

Open
imcom opened this issue Jul 1, 2021 · 10 comments
Open

runtime: unexpected return pc for golang.gopark #47003

imcom opened this issue Jul 1, 2021 · 10 comments

Comments

@imcom
Copy link

@imcom imcom commented Jul 1, 2021

What version of Go are you using (go version)?

$ go version
1.16

Does this issue reproduce with the latest release?

Did not try

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build814691616=/tmp/go-build -gno-record-gcc-switches"

uname -a

Linux USNATGW01 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

What did you do?

I was running gobgpd:v2.23 in production environment and I have no clue how to troubleshoot or reproduce this issue.
This only happened once in my production environment, no applicable logs when the crash happened

What did you expect to see?

I do not see why there is this unexpected return pc fatal error. My main concern is that since I do not know why it happened, I cannot tell when it will happen again if ever. Also I need help on interpreting the error dump and how to troubleshoot a runtime error like this. Thanks in advance.

What did you see instead?

runtime: unexpected return pc for runtime.gopark called from 0xc00a1946f0
stack: frame={sp:0xc000e694c8, fp:0xc000e694e8} stack=[0xc000e68000,0xc000e6a000)
000000c000e693c8:  0000000000000000  0000000000000000 
000000c000e693d8:  000000c000d3f400  000000000048b4dc  
000000c000e693e8:  000000c0000428d8  000000c000190800 
000000c000e693f8:  0000000000000004  000000c000d3f428 
000000c000e69408:  000000000075e9a7   000000c0000428d8 
000000c000e69418:  0000000000d02820  000000c0042ae000 
000000c000e69428:  000000c000d3f470  000000000075ec4c  
000000c000e69438:  000000c0000428a0  000000c0042ae000 
000000c000e69448:  000000c001497ae0  0000000000000010 
000000c000e69458:  000000c0042ae000  000000c0000428a0 
000000c000e69468:  0000000000d4e9e8  000000c000d3fe58 
000000c000e69478:  0000000000b21f45   000000c001497ae0 
000000c000e69488:  000000c000d3f588  0000000000000001 
000000c000e69498:  0000000000000001  000000c000d3f9c0 
000000c000e694a8:  0000000000000000  0000000000000000 
000000c000e694b8:  0000000000000000  0000000000000000 
000000c000e694c8: <000000c009302408  000000c00483f230 
000000c000e694d8:  000000c007c172d8 !000000c00a1946f0 
000000c000e694e8: >0000000000e238c8  000000c00d9b66d8 
000000c000e694f8:  000000c00a8af8c0  0000000000000013 
000000c000e69508:  0000000000000013  000000c004e5fce0 
000000c000e69518:  000000c0073908b8  000000c00e24e468 
000000c000e69528:  000000c000d3ff10  000000c00045b2c0 
000000c000e69538:  000000c00a079008  000000c000464c40 
000000c000e69548:  000000c00045b2b0  000000c000f7e420 
000000c000e69558:  000000c0074ac7f8  000000c000139968 
000000c000e69568:  000000c0035c1fc8  000000c000caefb0 
000000c000e69578:  0000000000000000  0000000000000000 
000000c000e69588:  0000000000bba9a0  0000000000df9f50 
000000c000e69598:  000000c00dcce078  000000c0030e15d8 
000000c000e695a8:  000000c00d8b3758  000000c009d647e0 
000000c000e695b8:  000000c00901eae0  000000c00f5aff38 
000000c000e695c8:  000000c00853ae88  000000c007897a28 
000000c000e695d8:  000000c00de05e78  000000c011ce25a0 
fatal error: unknown caller pc
fatal error: unexpected signal during runtime execution
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000e20dc0, fp:0xc000e20de0} stack=[0xc000e20800,0xc000e21000)
000000c000e20cc0:  0000000000000000  0000000000000000 
000000c000e20cd0:  0000000000000000  0000000000000000 
000000c000e20ce0:  0000000000000000  0000000000000000 
000000c000e20cf0:  0000000000000000  0000000000000000 
000000c000e20d00:  0000000000000000  0000000000000000 
000000c000e20d10:  0000000000000000  0000000000000000 
000000c000e20d20:  0000000000000000  0000000000000000 
000000c000e20d30:  0000000000000000  0000000000000000 
000000c000e20d40:  0000000000000000  0000000000000000 
000000c000e20d50:  0000000000000000  0000000000000000 
000000c000e20d60:  0000000000000000  0000000000000000 
000000c000e20d70:  0000000000000000  0000000000000000 
000000c000e20d80:  0000000000000000  0000000000000000 
000000c000e20d90:  0000000000000000  0000000000000000 
000000c000e20da0:  0000000000000000  0000000000000000 
000000c000e20db0:  0000000000000000  0000000000000000 
000000c000e20dc0: <0000000000000000  0000000000000000 
000000c000e20dd0:  0000000000000000 !0000000000000000 
000000c000e20de0: >0000000000000000  0000000000000000 
000000c000e20df0:  0000000000000000  0000000000000000 
000000c000e20e00:  0000000000000000  0000000000000000 
000000c000e20e10:  0000000000000000  0000000000000000 
000000c000e20e20:  0000000000000000  0000000000000000 
000000c000e20e30:  0000000000000000  0000000000000000 
000000c000e20e40:  0000000000000000  0000000000000000 
000000c000e20e50:  0000000000000000  0000000000000000 
000000c000e20e60:  0000000000000000  0000000000000000 
000000c000e20e70:  0000000000000000  0000000000000000 
000000c000e20e80:  0000000000000000  0000000000000000 
000000c000e20e90:  0000000000000000  0000000000000000 
000000c000e20ea0:  0000000000000000  0000000000000000 
000000c000e20eb0:  0000000000000000  0000000000000000 
000000c000e20ec0:  0000000000000000  0000000000000000 
000000c000e20ed0:  0000000000000000  0000000000000000 
fatal error: unknown caller pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000e5d4b8, fp:0xc000e5d4d8} stack=[0xc000e5c000,0xc000e5e000)
000000c000e5d3b8:  0000000000000000  0000000000000000 
000000c000e5d3c8:  0000000000000000  0000000000000000 
000000c000e5d3d8:  0000000000000000  0000000000000000 
000000c000e5d3e8:  0000000000000000  0000000000000000 
000000c000e5d3f8:  0000000000000000  0000000000000000 
000000c000e5d408:  0000000000000000  0000000000000000 
000000c000e5d418:  0000000000000000  0000000000000000 
000000c000e5d428:  0000000000000000  0000000000000000 
000000c000e5d438:  0000000000000000  0000000000000000 
000000c000e5d448:  0000000000000000  0000000000000000 
000000c000e5d458:  0000000000000000  0000000000000000 
000000c000e5d468:  0000000000000000  0000000000000000 
000000c000e5d478:  0000000000000000  0000000000000000 
000000c000e5d488:  0000000000000000  0000000000000000 
000000c000e5d498:  0000000000000000  0000000000000000 
000000c000e5d4a8:  0000000000000000  0000000000000000 
000000c000e5d4b8: <0000000000000000  0000000000000000 
000000c000e5d4c8:  0000000000000000 !0000000000000000 
000000c000e5d4d8: >0000000000000000  0000000000000000 
000000c000e5d4e8:  0000000000000000  0000000000000000 
000000c000e5d4f8:  0000000000000000  0000000000003023 
000000c000e5d508:  0000000000e23e58  000000c000396600 
000000c000e5d518:  0000000000000000  0000000000e1b280 
000000c000e5d528:  000000c01468d350  000000c0013ec520 
000000c000e5d538:  0000000000000000  000000c000a55920 
000000c000e5d548:  000000c002f53a00  000000000000001f 
000000c000e5d558:  0000000000000000  0000000000000000 
000000c000e5d568:  0000000000000000  0000000000000000 
000000c000e5d578:  000000c012edf680  0000000000e09f40 
000000c000e5d588:  000000c01468d380  000000c000623560 
000000c000e5d598:  000000c000b75700  000000c0102fd140 
000000c000e5d5a8:  0000000000000000  0000000000000000 
000000c000e5d5b8:  
@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 1, 2021

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

@imcom
Copy link
Author

@imcom imcom commented Jul 5, 2021

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

yes I've tried using the race-detector and there is no complaints from the tests

Just to make sure I merely need to run go test -race <target pkg> right ? Nothing stood out from the test.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 5, 2021

The race detector can only detect races if your test code exercises them. The litmus test is to pass -race to go build and deploy the application.

@imcom
Copy link
Author

@imcom imcom commented Jul 6, 2021

@davecheney oh, thanks, that's something I missed out, I will deploy the race-enabled version to see if we can catch anything and report back

@imcom
Copy link
Author

@imcom imcom commented Jul 17, 2021

Unfortunately ... enabled race detector does not give me more information regarding the issue.

runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000767190, fp:0xc0007671b0} stack=[0xc000766000,0xc000768000)
000000c000767090:  000000c0008947c8  0000000000000018
000000c0007670a0:  00000000011acf00  0000000000000000
000000c0007670b0:  0000000000000000  0000000000000000
000000c0007670c0:  0000000000000000  0000000000000000
000000c0007670d0:  0000000000000000  0000000000000000
000000c0007670e0:  000000c000c46000  0000000000000000
000000c0007670f0:  0000000000000000  0000000000000000
000000c000767100:  0000000000000000  0000000000000000
000000c000767110:  0000010100000000  0000000000000000
000000c000767120:  000000c000aa1b60  0000000000000000
000000c000767130:  000000c000b26180  0000000000000000
000000c000767140:  000000c000480600  0000000000000000
000000c000767150:  0000000000000000  0000000000000000
000000c000767160:  0000000000000000  0000000000000000
000000c000767170:  0000010000000000  0000000000000000
000000c000767180:  0000000000000000  0000000000000000
000000c000767190: <0000000000000000  0000000000000000
000000c0007671a0:  000000c000480600 !0000000000000000
000000c0007671b0: >0000000000000000  0000000000000000
000000c0007671c0:  0000000000000000  0000000000000000
000000c0007671d0:  0000000000000000  0000000000000000
000000c0007671e0:  0000000000000000  0000000000000000
000000c0007671f0:  0000000000000000  0000000000000000
000000c000767200:  000000c000480600  0000000000000000
000000c000767210:  0000000000000000  0000000000000000
000000c000767220:  0000000000000000  0000000000000000
000000c000767230:  0000000000000000  0000000000000000
000000c000767240:  0000000000000000  0000000000000000
000000c000767250:  0000000000000000  0000000000000000
000000c000767260:  000000c00037b500  0000000000000000
000000c000767270:  0000000000000000  000000c000a49f88
000000c000767280:  0000000000000000  0000000000000000
000000c000767290:  0000010100000000  0000000000000000
000000c0007672a0:  000000c000c02de0  0000000000000000
fatal error: unknown caller pc
runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unknown pc in defer 0x1
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x1e
stack: frame={sp:0xc00073f190, fp:0xc00073f1b0} stack=[0x5,0x1056580)

fatal error: unknown caller pc

@networkimprov
Copy link

@networkimprov networkimprov commented Jul 26, 2021

Since you can reproduce the issue, can you provide complete instructions to let others do so?

@imcom
Copy link
Author

@imcom imcom commented Aug 12, 2021

@networkimprov sorry for the late reply, I cannot really reproduce it , I just keep it running in our Staging environment, and the occurrence seems random to me. In addition to enable race on the build, I've enabled core dump this time, hope I can get more info and share here later. Just my feeling, it could be a runtime issue with memory management

@imcom
Copy link
Author

@imcom imcom commented Aug 13, 2021

Luckily I caught a coredump this time, but the backtrace is mysterious to me also

(gdb) where
#0  0x0000000000462a81 in runtime.sighandler (sig=0, info=0xc000360180, ctxt=0xc0004e9bf0, gp=0xc0004e9ac0) at /usr/local/go/src/runtime/signal_unix.go:593
#1  0x0000000000000000 in ?? ()
(gdb) bt full
#0  0x0000000000462a81 in runtime.sighandler (sig=0, info=0xc000360180, ctxt=0xc0004e9bf0, gp=0xc0004e9ac0) at /usr/local/go/src/runtime/signal_unix.go:593
        flags = 1902473840
        _g_ = 0x7270697271657270
        c = <optimized out>
        level = <optimized out>
#1  0x0000000000000000 in ?? ()
No symbol table info available.

@imcom
Copy link
Author

@imcom imcom commented Aug 13, 2021

Corresponding stderr goes like this

runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000767190, fp:0xc0007671b0} stack=[0xc000766000,0xc000768000)
000000c000767090:  000000c0008947c8  0000000000000018
000000c0007670a0:  00000000011acf00  0000000000000000
000000c0007670b0:  0000000000000000  0000000000000000
000000c0007670c0:  0000000000000000  0000000000000000
000000c0007670d0:  0000000000000000  0000000000000000
000000c0007670e0:  000000c000c46000  0000000000000000
000000c0007670f0:  0000000000000000  0000000000000000
000000c000767100:  0000000000000000  0000000000000000
000000c000767110:  0000010100000000  0000000000000000
000000c000767120:  000000c000aa1b60  0000000000000000
000000c000767130:  000000c000b26180  0000000000000000
000000c000767140:  000000c000480600  0000000000000000
000000c000767150:  0000000000000000  0000000000000000
000000c000767160:  0000000000000000  0000000000000000
000000c000767170:  0000010000000000  0000000000000000
000000c000767180:  0000000000000000  0000000000000000
000000c000767190: <0000000000000000  0000000000000000
000000c0007671a0:  000000c000480600 !0000000000000000
000000c0007671b0: >0000000000000000  0000000000000000
000000c0007671c0:  0000000000000000  0000000000000000
000000c0007671d0:  0000000000000000  0000000000000000
000000c0007671e0:  0000000000000000  0000000000000000
000000c0007671f0:  0000000000000000  0000000000000000
000000c000767200:  000000c000480600  0000000000000000
000000c000767210:  0000000000000000  0000000000000000
000000c000767220:  0000000000000000  0000000000000000
000000c000767230:  0000000000000000  0000000000000000
000000c000767240:  0000000000000000  0000000000000000
000000c000767250:  0000000000000000  0000000000000000
000000c000767260:  000000c00037b500  0000000000000000
000000c000767270:  0000000000000000  000000c000a49f88
000000c000767280:  0000000000000000  0000000000000000
000000c000767290:  0000010100000000  0000000000000000
000000c0007672a0:  000000c000c02de0  0000000000000000
fatal error: unknown caller pc
runtime: unknown pc in defer 0x0
fatal error: unknown pc
runtime: unknown pc in defer 0x1
fatal error: unknown pc
runtime: unexpected return pc for runtime.gopark called from 0x1e
stack: frame={sp:0xc00073f190, fp:0xc00073f1b0} stack=[0x5,0x1056580)

fatal error: unknown caller pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc0006729f0, fp:0xc000672a10} stack=[0xc000670000,0xc000674000)
000000c0006728f0:  0000000000000000  0000000000000000
000000c000672900:  0000000000000000  0000000000000000
000000c000672910:  0000000000000000  ffffffffffffffff
000000c000672920:  0000000000000000  0000000000000000
000000c000672930:  0000000000000000  000000c00a5eff80
000000c000672940:  0000000000454d12 <runtime.goexit1+114>  000000c000c2c900
000000c000672950:  0000000000000000  0000000000000000
000000c000672960:  0000000000000000  000000c00a5effa8
000000c000672970:  0000000000000000  0000000000503865 <syscall.Syscall+5>
000000c000672980:  000000c00a5effb8  0000000000000000
000000c000672990:  0000000000000006  000000000014fe82
000000c0006729a0:  000000c002159800  0000000000000000
000000c0006729b0:  0000000000000000  0000000001000000
000000c0006729c0:  0000000000000000  0000000000000000
000000c0006729d0:  0000000000000000  0000000000000000
000000c0006729e0:  0000000000000000  0000000000000000
000000c0006729f0: <0000000000000000  0000000000000000
000000c000672a00:  0000000000000000 !0000000000000000
000000c000672a10: >0000000000000000  0000000000f6445e <github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer+126>
000000c000672a20:  0000000000000000  0000000000f94f20 <github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer.func1+0>
000000c000672a30:  00007fec4bfa0008  0000000000000000
000000c000672a40:  0000000000000000  0000000000000000
000000c000672a50:  0000000000000000  0000000000000000
000000c000672a60:  0000000000000000  0000000000000000
000000c000672a70:  0000000000000000  0000000000000000
000000c000672a80:  0000000000000000  0000000000000000
000000c000672a90:  000000c00499cba0  ffffffffffffffff
000000c000672aa0:  0000000000000000  0000000000000000
000000c000672ab0:  0000000000000000  000000c00499cfa0
000000c000672ac0:  0000000000454d12 <runtime.goexit1+114>  000000c000c2ca80
000000c000672ad0:  0000000000000000  0000000000000000
000000c000672ae0:  0000000000000000  000000c00499cfc8
000000c000672af0:  0000000000000000  0000000000503865 <syscall.Syscall+5>
000000c000672b00:  000000c00499cfd8  0000000000000000
fatal error: unknown caller pc
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc00066d190, fp:0xc00066d1b0} stack=[0xc00066c000,0xc00066e000)
000000c00066d090:  000000c0000d07e0  000000c000d60a50
000000c00066d0a0:  c03c52e076eedf23  runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc000403190, fp:0xc0004031b0} stack=[0xc000402000,0xc000404000)
000000c000403090:  0000000000000000  0000000000000000
000000c0004030a0:  0000000000000000  0000000000000000
000000c0004030b0:  0000000000000000  0000000000000000
000000c0004030c0:  0000000000000000  0000000000000000
000000c0004030d0:  0000000000000000  0000000000000000
000000c0004030e0:  0000000000000000  0000000000000000
000000c0004030f0:  0000000000000000  0000000000000000
000000c000403100:  0000000000000000  0000000000000000
000000c000403110:  0000000000000000  0000000000000000
000000c000403120:  0000000000000000  0000000000000000
000000c000403130:  0000000000000000  0000000000000000
000000c000403140:  0000000000000000  0000000000000000
000000c000403150:  0000000000000000  0000000000000000
000000c000403160:  0000000000000000  0000000000000000
000000c000403170:  0000000000000000  0000000000000000
000000c000403180:  0000000000000000  0000000000000000
000000c000403190: <0000000000000000  0000000000000000
000000c0004031a0:  0000000000000000 !0000000000000000
000000c0004031b0: >0000000000000000  0000000000000000
000000c0004031c0:  0000000000000000  0000000000000000
000000c0004031d0:  0000000000000000  0000000000000000
000000c0004031e0:  0000000000000000  0000000000000000
000000c0004031f0:  0000000000000000  0000000000000000
000000c000403200:  0000000000000000  0000000000000000
000000c000403210:  0000000000000000  0000000000000000
000000c000403220:  0000000000000000  0000000000000000
000000c000403230:  0000000000571080 <context.WithCancel.func1+0>  000000c000e77a00
000000c000403240:  0000000000000000  0000000000000000
000000c000403250:  0000000000000000  0000000000000000
000000c000403260:  0000000000000000  0000000000000000
000000c000403270:  0000000000000000  0000000000000000
000000c000403280:  0000000000000000  0000000000000000
000000c000403290:  0000000000000000  0000000000000000
000000c0004032a0:  0000000000000000  0000000000000000
runtime: pointer 0xc000959200 to unallocated span span.base()=0xc0000fe000 span.limit=0xc0000fff80 span.state=0
runtime: found in object at *(0xc00037ed00+0x2d0)
object=0xc00037ed00 s.base()=0xc00037c000 s.limit=0xc00037ff00 s.spanclass=78 s.elemsize=2304 s.state=mSpanInUse
 *(object+0) = 0x4
 *(object+8) = 0xc000af6ab0
 *(object+16) = 0x4
 *(object+24) = 0x4
 *(object+32) = 0xc000a697e0
 *(object+40) = 0x4
 *(object+48) = 0x4
 *(object+56) = 0xc000af6ac0
 *(object+64) = 0x4
 *(object+72) = 0x4
 *(object+80) = 0xc000a697e0
 *(object+88) = 0x4
 *(object+96) = 0x4
 *(object+104) = 0xc000af6ad0

@imcom
Copy link
Author

@imcom imcom commented Aug 13, 2021

Output from dlv

0  0x0000000000462a81 in runtime.sighandler
   at /usr/local/go/src/runtime/signal_unix.go:593
(dlv) ls
> runtime.sighandler() /usr/local/go/src/runtime/signal_unix.go:593 (PC: 0x462a81)
Warning: debugging optimized function
   588:
   589:			c.preparePanic(sig, gp)
   590:			return
   591:		}
   592:
=> 593:		if c.sigcode() == _SI_USER || flags&_SigNotify != 0 {
   594:			if sigsend(sig) {
   595:				return
   596:			}
   597:		}
   598:
(dlv)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants