Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: unexpected signal during runtime execution on Debian #29377

Open
cosmtrek opened this issue Dec 21, 2018 · 2 comments
Open

runtime: unexpected signal during runtime execution on Debian #29377

cosmtrek opened this issue Dec 21, 2018 · 2 comments
Labels
NeedsInvestigation
Milestone
Backlog

Comments

@cosmtrek
Copy link

@cosmtrek cosmtrek commented Dec 21, 2018

What version of Go are you using (go version)?

$ go version
go1.10.1 linux/amd64

Does this issue reproduce with the latest release?

No

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tiger/.cache/"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/tiger/compile_path"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build590147109=/tmp/go-build -gno-record-gcc-switches"

What did you do?

My service started in a docker container and happened to throw panic. I cannot figure it out.

The stack shows as follow:

fatal error: unexpected signal during runtime execution
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x34 pc=0x4341f1]

  runtime stack:
  runtime: unexpected return pc for runtime.schedule called from 0x100000002
  stack: frame={sp:0x7fa5c902ed78, fp:0x7fa5c902edc0} stack=[0x7fa5c882f2b0,0x7fa5c902eeb0)
  00007fa5c902ec78:  0000000000000034  0000000000000001
  00007fa5c902ec88:  000000000249e56c  0000000000000000
  00007fa5c902ec98:  00007fa5c902ecc0  0000000000457b8c <runtime.dopanic.func1+60>
  00007fa5c902eca8:  000000c4200f0600  000000000042c921 <runtime.throw+129>
  00007fa5c902ecb8:  00007fa5c902ed08  00007fa5c902ecf8
  00007fa5c902ecc8:  000000000042c84a <runtime.dopanic+74>  00007fa5c902ecd8
  00007fa5c902ecd8:  0000000000457b50 <runtime.dopanic.func1+0>  000000c4200f0600
  00007fa5c902ece8:  000000000042c921 <runtime.throw+129>  00007fa5c902ed08
  00007fa5c902ecf8:  00007fa5c902ed18  000000000042c921 <runtime.throw+129>
  00007fa5c902ed08:  0000000000000000  000000000000002a
  00007fa5c902ed18:  00007fa5c902ed68  00000000004423be <runtime.sigpanic+654>
  00007fa5c902ed28:  00000000024f5aac  000000000000002a
  00007fa5c902ed38:  0000000000439fd7 <runtime.runqsteal+87>  000000c420050a00
  00007fa5c902ed48:  000000c42004c5e8  000000c4200f0600
  00007fa5c902ed58:  0000000000000000  000b947b2004c000
  00007fa5c902ed68:  00007fa5c902ee10  00000000004341f1 <runtime.schedule+225>
  00007fa5c902ed78: <000000c42004c000  000000c420050a00
  00007fa5c902ed88:  000000c400000000  0000000000000000
  00007fa5c902ed98:  00007fa5c902edd0  0000000000000004
  00007fa5c902eda8:  0000000000000000  0000000400000003
  00007fa5c902edb8: !0000000100000002 >000000c42004c000
  00007fa5c902edc8:  000000c4201f6800  00007fa5c902edf0
  00007fa5c902edd8:  0000000000432bc8 <runtime.wakep+72>  0000000000000000
  00007fa5c902ede8:  000000c42004a000  00000000004331c5 <runtime.execute+261>
  00007fa5c902edf8:  000000c4200f0600  000000c42004c000
  00007fa5c902ee08:  00007fa5c902ed88  00007fa5c902ee58
  00007fa5c902ee18:  000000000043424b <runtime.schedule+315>  000000c42004c000
  00007fa5c902ee28:  0000000000000000  0000000000436300 <runtime.gfput+256>
  00007fa5c902ee38:  00007fa5c902ee58  000000000043d435 <runtime.selparkcommit+133>
  00007fa5c902ee48:  000000c4200f0600  000000c4222fa120
  00007fa5c902ee58:  00007fa5c902ee88  0000000000434576 <runtime.park_m+182>
  00007fa5c902ee68:  000000c4201f2a80  0000000000000000
  00007fa5c902ee78:  00007fa5cc7d0601  000000c4200f0600
  00007fa5c902ee88:  000000c421a4e268  000000000045925b <runtime.mcall+91>
  00007fa5c902ee98:  000000c4201f2a80  0000000000000000
  00007fa5c902eea8:  0100000000000000
  runtime.throw(0x24f5aac, 0x2a)
      /usr/local/go/src/runtime/panic.go:616 +0x81
  runtime.sigpanic()
      /usr/local/go/src/runtime/signal_unix.go:372 +0x28e
  runtime: unexpected return pc for runtime.schedule called from 0x100000002
  stack: frame={sp:0x7fa5c902ed78, fp:0x7fa5c902edc0} stack=[0x7fa5c882f2b0,0x7fa5c902eeb0)
  00007fa5c902ec78:  0000000000000034  0000000000000001
  00007fa5c902ec88:  000000000249e56c  0000000000000000
  00007fa5c902ec98:  00007fa5c902ecc0  0000000000457b8c <runtime.dopanic.func1+60>
  00007fa5c902eca8:  000000c4200f0600  000000000042c921 <runtime.throw+129>
  00007fa5c902ecb8:  00007fa5c902ed08  00007fa5c902ecf8
  00007fa5c902ecc8:  000000000042c84a <runtime.dopanic+74>  00007fa5c902ecd8
  00007fa5c902ecd8:  0000000000457b50 <runtime.dopanic.func1+0>  000000c4200f0600
  00007fa5c902ece8:  000000000042c921 <runtime.throw+129>  00007fa5c902ed08
  00007fa5c902ecf8:  00007fa5c902ed18  000000000042c921 <runtime.throw+129>
  00007fa5c902ed08:  0000000000000000  000000000000002a
  00007fa5c902ed18:  00007fa5c902ed68  00000000004423be <runtime.sigpanic+654>
  00007fa5c902ed28:  00000000024f5aac  000000000000002a
  00007fa5c902ed38:  0000000000439fd7 <runtime.runqsteal+87>  000000c420050a00
  00007fa5c902ed48:  000000c42004c5e8  000000c4200f0600
  00007fa5c902ed58:  0000000000000000  000b947b2004c000
  00007fa5c902ed68:  00007fa5c902ee10  00000000004341f1 <runtime.schedule+225>
  00007fa5c902ed78: <000000c42004c000  000000c420050a00
  00007fa5c902ed88:  000000c400000000  0000000000000000
  00007fa5c902ed98:  00007fa5c902edd0  0000000000000004
  00007fa5c902eda8:  0000000000000000  0000000400000003
  00007fa5c902edb8: !0000000100000002 >000000c42004c000
  00007fa5c902edc8:  000000c4201f6800  00007fa5c902edf0
  00007fa5c902edd8:  0000000000432bc8 <runtime.wakep+72>  0000000000000000
  00007fa5c902ede8:  000000c42004a000  00000000004331c5 <runtime.execute+261>
  00007fa5c902edf8:  000000c4200f0600  000000c42004c000
  00007fa5c902ee08:  00007fa5c902ed88  00007fa5c902ee58
  00007fa5c902ee18:  000000000043424b <runtime.schedule+315>  000000c42004c000
  00007fa5c902ee28:  0000000000000000  0000000000436300 <runtime.gfput+256>
  00007fa5c902ee38:  00007fa5c902ee58  000000000043d435 <runtime.selparkcommit+133>
  00007fa5c902ee48:  000000c4200f0600  000000c4222fa120
  00007fa5c902ee58:  00007fa5c902ee88  0000000000434576 <runtime.park_m+182>
  00007fa5c902ee68:  000000c4201f2a80  0000000000000000
  00007fa5c902ee78:  00007fa5cc7d0601  000000c4200f0600
  00007fa5c902ee88:  000000c421a4e268  000000000045925b <runtime.mcall+91>
  00007fa5c902ee98:  000000c4201f2a80  0000000000000000
  00007fa5c902eea8:  0100000000000000
  runtime.schedule()
      /usr/local/go/src/runtime/proc.go:2547 +0xe1

// some goroutines maybe helpful to show what happened
goroutine 1 [select, 11 minutes]:
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.waitSignal(0xc42239f140, 0x0, 0x0)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:138 +0x1cf
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.Run(0xc400000018, 0x25ed2b8)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:126 +0x66
  main.main()
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/server.go:78 +0x5d6

  goroutine 5 [select]:
  code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics.(*Sender).emitCounterLoop(0xc4200e2100)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics/sender.go:221 +0x167
  created by code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics.(*Sender).runLoops
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics/sender.go:213 +0x3f

...

goroutine 2268 [IO wait]:
  internal/poll.runtime_pollWait(0x7fa5cc7846a0, 0x72, 0x0)
      /usr/local/go/src/runtime/netpoll.go:173 +0x57
  internal/poll.(*pollDesc).wait(0xc4201d8998, 0x72, 0xc4222bf200, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
  internal/poll.(*pollDesc).waitRead(0xc4201d8998, 0xffffffffffffff00, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
  internal/poll.(*FD).Accept(0xc4201d8980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
  net.(*netFD).accept(0xc4201d8980, 0xa17e614831, 0x3b8c160, 0x1)
      /usr/local/go/src/net/fd_unix.go:238 +0x42
  net.(*TCPListener).accept(0xc4202be3a8, 0x458530, 0xc42d0abe38, 0xc42d0abe40)
      /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
  net.(*TCPListener).Accept(0xc4202be3a8, 0x25ed4a8, 0xc420da2f00, 0x26f0d00, 0xc440373ad8)
      /usr/local/go/src/net/tcpsock.go:259 +0x49
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.(*RpcServer).AcceptLoop(0xc420da2f00, 0x0, 0x0)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite_server.go:106 +0x4b
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.(*RpcServer).Serve(0xc420da2f00, 0x0, 0x0)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite_server.go:189 +0x118
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.Run.func1(0xc42239f140)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:124 +0x2d
  created by code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.Run
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:123 +0x58

...

goroutine 2277 [IO wait, 11 minutes]:
  internal/poll.runtime_pollWait(0x7fa5cc784910, 0x72, 0x0)
      /usr/local/go/src/runtime/netpoll.go:173 +0x57
  internal/poll.(*pollDesc).wait(0xc420d7f398, 0x72, 0xc422944f00, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
  internal/poll.(*pollDesc).waitRead(0xc420d7f398, 0xffffffffffffff00, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
  internal/poll.(*FD).Accept(0xc420d7f380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
  net.(*netFD).accept(0xc420d7f380, 0x7fa5cc7d0d90, 0x0, 0x25f12c0)
      /usr/local/go/src/net/fd_unix.go:238 +0x42
  net.(*TCPListener).accept(0xc4201bc128, 0xc4229b1e10, 0x412758, 0x30)
      /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
  net.(*TCPListener).AcceptTCP(0xc4201bc128, 0x2235680, 0xc421a30cc0, 0x210ab60)
      /usr/local/go/src/net/tcpsock.go:246 +0x49
  net/http.tcpKeepAliveListener.Accept(0xc4201bc128, 0xc420044110, 0x210ab60, 0x3b6f970, 0x23954e0)
      /usr/local/go/src/net/http/server.go:3216 +0x2f
  net/http.(*Server).Serve(0xc420106f70, 0x26ecbc0, 0xc4201bc128, 0x0, 0x0)
      /usr/local/go/src/net/http/server.go:2770 +0x1a5
  net/http.(*Server).ListenAndServe(0xc420106f70, 0xc420106f70, 0x1)
      /usr/local/go/src/net/http/server.go:2711 +0xa9
  net/http.ListenAndServe(0xc420eaafb8, 0x6, 0x0, 0x0, 0x1, 0x0)
      /usr/local/go/src/net/http/server.go:2969 +0x7a
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.startDebugServer.func2()
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/debug.go:40 +0xb2
  created by code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.startDebugServer
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/debug.go:37 +0x101

I suspect some goroutines waiting for too long to make runtime scheduler abnormal, but I cannot reproduce it.
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Dec 21, 2018

Is this repeatable?

CC @aclements

@cosmtrek
Copy link
Author

@cosmtrek cosmtrek commented Dec 21, 2018

@ianlancetaylor No, rarely. The difficulty is I could not reproduce it.

@katiehockman katiehockman added the NeedsInvestigation label Jan 2, 2019
@katiehockman katiehockman added this to the Go1.13 milestone Jan 2, 2019
@andybons andybons removed this from the Go1.13 milestone Jul 8, 2019
@andybons andybons added this to the Go1.14 milestone Jul 8, 2019
@rsc rsc removed this from the Go1.14 milestone Oct 9, 2019
@rsc rsc added this to the Backlog milestone Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
5 participants
@rsc @andybons @ianlancetaylor @cosmtrek @katiehockman