runtime: unexpected signal during runtime execution #41099

dnephin · 2020-08-28T17:02:40Z

What version of Go are you using (`go version`)?

go1.14.7, also observed the same problem with earlier versions: go1.13.7

Does this issue reproduce with the latest release?

Unknown

What operating system and processor architecture are you using (`go env`)?

Linux 4.14.35-1818.3.3.el7uek.x86_64 #2 SMP Mon Sep 24 14:45:01 PDT 2018 x86_64 x86_64 x86_64 GNU/Linux
(Oracle Linux Server release 7.6)

What did you do?

Original issue reported here: hashicorp/consul#8558

The Consul binary appears to be crashing with errors from the runtime, ex:

fatal error: stopg: invalid status
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
runtime: unexpected return pc for runtime.sigpanic called from 0x0

The stack trace from the application shows the line as a select blocking on a channel from signal.Notify (source). I'm not sure if that is related to the underlying issue, or has to do with the signal from the runtime.

Full trace

runtime: gp: gp=0xc000450180, goid=0, gp->atomicstatus=0
runtime:  g:  g=0xc0005b5680, goid=0,  g->atomicstatus=0
fatal error: stopg: invalid status
runtime: gp: gp=0xc000450300, goid=0, gp->atomicstatus=0
runtime:  g:  g=0xc000604180, goid=0,  g->atomicstatus=0
fatal error: unexpected signal during runtime execution
fatal error: stopg: invalid status
runtime: gp: gp=0xc000450000, goid=0, gp->atomicstatus=0
runtime:  g:  g=0xc000566180, goid=0,  g->atomicstatus=0
fatal error: stopg: invalid status
runtime stack:
runtime.throw(0x2fac299, 0x15)
/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.scang(0xc000450180, 0xc000060170)
/usr/local/go/src/runtime/proc.go:869 +0x379
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:221 +0x6d
runtime.markroot(0xc000060170, 0xc00000001b)
/usr/local/go/src/runtime/mgcmark.go:202 +0x2f3
runtime.gcDrain(0xc000060170, 0x7)
/usr/local/go/src/runtime/mgcmark.go:915 +0x110
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1936 +0x16b
runtime.systemstack(0x0)
/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1146
goroutine 34 [GC worker (idle)]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc00028ef60 sp=0xc00028ef58 pc=0x45d6d0
runtime.gcBgMarkWorker(0xc00005ef00)
/usr/local/go/src/runtime/mgc.go:1900 +0x1bd fp=0xc00028efd8 sp=0xc00028ef60 pc=0x41c38d
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00028efe0 sp=0xc00028efd8 pc=0x45f621
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1794 +0x77
goroutine 1 [select, 149 minutes]:
github.com/hashicorp/consul/command/agent.(*cmd).run(0xc000651500, 0xc000170020, 0x8, 0x8, 0x0)
/home/circleci/project/consul/command/agent/agent.go:331 +0x13eb
github.com/hashicorp/consul/command/agent.(*cmd).Run(0xc000651500, 0xc000170020, 0x8, 0x8, 0xc0001452a0)
/home/circleci/project/consul/command/agent/agent.go:78 +0x4d
github.com/mitchellh/cli.(*CLI).Run(0xc00031c000, 0xc00031c000, 0x80, 0xc000145ac0)
/go/pkg/mod/github.com/mitchellh/cli@v1.1.0/cli.go:260 +0x1da
main.realMain(0xc00014a058)
/home/circleci/project/consul/main.go:53 +0x393
main.main()
/home/circleci/project/consul/main.go:20 +0x22
goroutine 20 [syscall, 149 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41
goroutine 23 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x50186a0)
/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x8b
created by github.com/golang/glog.init.0
/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x26f
goroutine 0 [idle]:
fatal error: unexpected signal during runtime execution
panic during panic
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x452486]
runtime stack:
runtime.throw(0x300b2ed, 0x2a)
/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:378 +0x47c
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000450000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/runtime/traceback.go:159 +0x156
runtime.traceback1(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000450000, 0x0)
/usr/local/go/src/runtime/traceback.go:722 +0xf0
runtime.traceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000450000)
/usr/local/go/src/runtime/traceback.go:676 +0x52
runtime.tracebackothers(0xc0005b5680)
/usr/local/go/src/runtime/traceback.go:929 +0x1a6
runtime.dopanic_m(0xc0005b5680, 0x42f602, 0xc000aabdf8, 0x1)
/usr/local/go/src/runtime/panic.go:974 +0x29e
runtime.fatalthrow.func1()
/usr/local/go/src/runtime/panic.go:829 +0x5f
runtime.fatalthrow()
/usr/local/go/src/runtime/panic.go:826 +0x57
runtime.throw(0x2fac299, 0x15)
/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.scang(0xc000450180, 0xc000060170)
/usr/local/go/src/runtime/proc.go:869 +0x379
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:221 +0x6d
runtime.markroot(0xc000060170, 0xc00000001b)
/usr/local/go/src/runtime/mgcmark.go:202 +0x2f3
runtime.gcDrain(0xc000060170, 0x7)
/usr/local/go/src/runtime/mgcmark.go:915 +0x110
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1936 +0x16b
runtime.systemstack(0x0)
/usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1146
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]
runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x0
stack: frame={sp:0xc000c61dd8, fp:0xc000c61e08} stack=[0xc000c60000,0xc000c62000)
000000c000c61cd8:  000000c0007bc180  01000000000003e8
000000c000c61ce8:  0000000000000004  000000000000001f
000000c000c61cf8:  0000000000000000  0000000000000000
000000c000c61d08:  0000000000000001  0000000002fdb0bd
000000c000c61d18:  000000c000c61d60  000000000045bfbf <runtime.fatalthrow.func1+95>

I looked around this issue tracker for an existing issue. I saw some related things, but nothing that sounded like exactly the same problem.

One issue mentioned C signal handlers. There should not be any such handlers in this binary.

The text was updated successfully, but these errors were encountered:

davecheney · 2020-08-28T17:07:17Z

Sadly, with the release of go 1.15, go 1.13 is no longer receiving active support.

Can you please try with the latest release of Go can confirm if the problem occurs.

dnephin · 2020-08-28T17:13:22Z

Ah, that's right, thank you. I've asked they try with a more recent release, which uses go1.14.x.

dnephin · 2020-09-08T17:11:24Z

More details from hashicorp/consul#8558 (comment). They were able to reproduce with go1.14.7. The full log is attached. They also said it occurs only on hosts with Intel Optane connected in RAM mode.

The stack is a bit different, but the errors look similar

fatal error: unexpected signal during runtime execution
fatal error: runtimer: bad p
runtime: unexpected return pc for runtime.sigpanic called from 0x0
fatal error: cleantimers: bad p

consul-agent-kn-0030_2020.09.02_trace.log

networkimprov · 2020-09-08T19:38:16Z

cc @aclements @randall77 @prattmic

@gopherbot remove WaitingForInfo

ianlancetaylor · 2020-09-08T21:38:50Z

This looks a lot like memory corruption.

fanatl · 2020-09-09T06:55:10Z

Hello to all.
These fatal errors have appeared since version 1.12 and older.
And is repeated only on hosts with Intel optane as RAM.

davecheney · 2020-09-09T07:17:59Z

All the more reason to determine if the binary contains data races or cgo related memory corruption.

networkimprov · 2020-09-09T18:20:40Z

@dnephin have you built the app with the race detector?

dnephin · 2020-09-09T18:28:18Z

I've been working on running the tests with the race detector enabled. We run some packages with -race, but not all. We don't build binaries with it enabled. Would that help identify the problem?

networkimprov · 2020-09-09T18:34:41Z

A data race could cause memory corruption, so building the app with the race detector may help.

woblerr · 2021-02-20T07:39:07Z

@dnephin any new about this problem?

blanchonvincent · 2021-05-16T17:29:14Z

@dnephin @woblerr I got the same issues in my code, and it broke since Go 1.14 with the introduction of the goroutine async preemption (I do not have control over the C code I am using, but it looks like it mess up with the signals), so I just disable the async preemption when running my tests/building the binary. Your logs suggests it could be the same issue (crash in the garbage collector that preempt the goroutines). Can you try running your tests with GODEBUG=asyncpreemptoff=1 like GODEBUG=asyncpreemptoff=1 go test ./... to see if it is the cause?

gopherbot · 2021-06-17T00:20:47Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

networkimprov · 2021-06-22T16:01:16Z

I previously proposed a way to disable async preemption in all builds of an app; see #40870

It was closed without action, but you might want to comment there to request reconsideration of the idea.

dnephin mentioned this issue Aug 28, 2020

fatal error: unexpected signal during runtime execution hashicorp/consul#8558

Open

cagedmantis changed the title ~~[go1.13.15] fatal error: unexpected signal during runtime execution~~ runtime: unexpected signal during runtime execution Aug 30, 2020

cagedmantis added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 30, 2020

gopherbot removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 8, 2020

andybons added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 8, 2020

andybons added this to the Unplanned milestone Sep 8, 2020

This was referenced Sep 9, 2020

runtime: "fatal error: unexpected signal during runtime execution" on windows-amd64-longtest builder of Go 1.15.2 commit #41296

Closed

runtime: unexpected signal during runtime execution #41285

Closed

woblerr mentioned this issue Dec 29, 2020

Stolon components fail with an error: unexpected signal during runtime execution sorintlab/stolon#816

Closed

egonelbre mentioned this issue Jan 4, 2021

runtime: unexpected return pc for runtime.sigpanic called from 0x1 #43496

Open

davecheney added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 17, 2021

gopherbot closed this as completed Jun 17, 2021

networkimprov mentioned this issue Jun 22, 2021

proposal: cmd/go: build option to set env variable defaults #40870

Closed

golang locked and limited conversation to collaborators Jun 22, 2022

gopherbot added the FrozenDueToAge label Jun 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: unexpected signal during runtime execution #41099

runtime: unexpected signal during runtime execution #41099

dnephin commented Aug 28, 2020 •

edited

davecheney commented Aug 28, 2020

dnephin commented Aug 28, 2020

dnephin commented Sep 8, 2020

networkimprov commented Sep 8, 2020

ianlancetaylor commented Sep 8, 2020

fanatl commented Sep 9, 2020

davecheney commented Sep 9, 2020

networkimprov commented Sep 9, 2020

dnephin commented Sep 9, 2020

networkimprov commented Sep 9, 2020

woblerr commented Feb 20, 2021

blanchonvincent commented May 16, 2021

gopherbot commented Jun 17, 2021

networkimprov commented Jun 22, 2021

runtime: unexpected signal during runtime execution #41099

runtime: unexpected signal during runtime execution #41099

Comments

dnephin commented Aug 28, 2020 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

davecheney commented Aug 28, 2020

dnephin commented Aug 28, 2020

dnephin commented Sep 8, 2020

networkimprov commented Sep 8, 2020

ianlancetaylor commented Sep 8, 2020

fanatl commented Sep 9, 2020

davecheney commented Sep 9, 2020

networkimprov commented Sep 9, 2020

dnephin commented Sep 9, 2020

networkimprov commented Sep 9, 2020

woblerr commented Feb 20, 2021

blanchonvincent commented May 16, 2021

gopherbot commented Jun 17, 2021

networkimprov commented Jun 22, 2021

dnephin commented Aug 28, 2020 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?