Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: unexpected return pc for runtime.gopark called from 0x1 #19955

t57root opened this issue Apr 13, 2017 · 8 comments


None yet
7 participants
Copy link

commented Apr 13, 2017

What version of Go are you using (go version)?

go version go1.7.4 linux/amd64

What operating system and processor architecture are you using (go env)?

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build295860934=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I use signal.Notify() to get a channel to notify me when SIGTERM/SIGINT occurred. But when I hit ctrl+c(which produce SIGINT), the program sometimes crashed, showing runtime: unexpected return pc for runtime.gopark called from 0x1. This error seems happened conditionally, not inevitably.
It's a large project and I don't know where exactly caused this problem. I kinda think such low-level error should not be caused by my user code. Here's the callstack printed automatically:

runtime: unexpected return pc for runtime.gopark called from 0x1
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x9e8032, 0x11)
    /usr/local/go/src/runtime/panic.go:566 +0x95
runtime.gentraceback(0xffffffffffffffff, 0xc4200236d8, 0x0, 0xc420001380, 0x0, 0x0, 0x7fffffff, 0x7f2d5e7fb8b8, 0x0, 0x0, ...)
    /usr/local/go/src/runtime/traceback.go:317 +0x13f6
runtime.scanstack(0xc420001380, 0xc42001b228)
    /usr/local/go/src/runtime/mgcmark.go:777 +0x259
runtime.scang(0xc420001380, 0xc42001b228)
    /usr/local/go/src/runtime/proc.go:828 +0x14b
    /usr/local/go/src/runtime/mgcmark.go:236 +0x6d
    /usr/local/go/src/runtime/asm_amd64.s:298 +0x79

goroutine 903 [running]:
    /usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420192698 sp=0xc420192690
runtime.markroot(0xc42001b228, 0xc40000000a)
    /usr/local/go/src/runtime/mgcmark.go:241 +0x2ba fp=0xc420192710 sp=0xc420192698
runtime.gcDrain(0xc42001b228, 0x5)
    /usr/local/go/src/runtime/mgcmark.go:963 +0xab fp=0xc420192748 sp=0xc420192710
    /usr/local/go/src/runtime/mgc.go:1452 +0x1d1 fp=0xc4201927b8 sp=0xc420192748
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4201927c0 sp=0xc4201927b8
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1341 +0x98

goroutine 1 [select]:
    /root/workspace/zagen/src/zagen/core/scheduler.go:257 +0x602
main.daemon(0xc4200e5550, 0x13)
    /root/workspace/zagen/src/zagen/daemon.go:65 +0x5b0
    /root/workspace/zagen/src/zagen/main.go:110 +0x645

goroutine 17 [syscall, 1 minutes, locked to thread]:
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 6 [syscall, 1 minutes]:
    /usr/local/go/src/runtime/sigqueue.go:116 +0x157
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 9 [chan receive]:
    /root/workspace/zagen/src/zagen/rpc/log.go:142 +0xa1
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:30 +0xf2

goroutine 10 [select, 1 minutes]:
    /root/workspace/zagen/src/zagen/rpc/json.go:202 +0x152
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:40 +0x1f8

goroutine 11 [select, 1 minutes]:
    /root/workspace/zagen/src/zagen/rpc/trans.go:78 +0x18f
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:46 +0x31c

goroutine 12 [select, 1 minutes]:
    /root/workspace/zagen/src/zagen/rpc/trans.go:78 +0x18f
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:46 +0x31c

goroutine 24 [chan receive]:
zagen/core.MonitorUpdate(0xe404a0, 0xc4200b81e0)
    /root/workspace/zagen/src/zagen/core/scheduler.go:243 +0x80
created by main.daemon
    /root/workspace/zagen/src/zagen/daemon.go:64 +0x5ab

goroutine 25 [select, 1 minutes, locked to thread]:
runtime.gopark(0xa2df80, 0x0, 0x9def92, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc42001f730, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:423 +0x11d9
    /usr/local/go/src/runtime/select.go:238 +0x1c
    /usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 448 [chan receive, 1 minutes]:
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 35 [runnable, locked to thread]:, 0xc42046c860, 0xc42046c868, 0x65) +0x4d*SQLiteStmt).exec(0xc4204b5530, 0x7f2d815061e8, 0xc4200122c8, 0xc4204bee10, 0x6, 0x6, 0x0, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/ +0x33a*SQLiteConn).exec(0xc42040d810, 0x7f2d815061e8, 0xc4200122c8, 0xc42014f260, 0x6c, 0xc4204bee10, 0x6, 0x6, 0x10, 0x4e2f01, ...)
    /root/workspace/zagen/src/ +0x2f9*SQLiteConn).Exec(0xc42040d810, 0xc42014f260, 0x6c, 0xc42015ac60, 0x6, 0x6, 0x6, 0x0, 0x0, 0xa2ad9e)
    /root/workspace/zagen/src/ +0x1f0
database/sql.(*DB).exec(0xc4200e4630, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x4bd37f01, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/database/sql/sql.go:1036 +0x4ca
database/sql.(*DB).Exec(0xc4200e4630, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x10, 0xc420049388, 0xc42046c810, 0xc4200493c8)
    /usr/local/go/src/database/sql/sql.go:1010 +0xa2*Session).innerExec(0xc42012b180, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x0, 0xc420049358, 0xa2ad7a, 0x10)
    /root/workspace/zagen/src/ +0x88*Session).exec.func1(0x10, 0x10, 0xc42046c820, 0x0)
    /root/workspace/zagen/src/ +0xe8*Engine).logSQLExecutionTime(0xc420255720, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0xc4200494d0, 0x6c, 0x0, 0x0, ...)
    /root/workspace/zagen/src/ +0x54*Session).exec(0xc42012b180, 0xc42014f260, 0x6c, 0xc42015aba0, 0x6, 0x6, 0x6c, 0x6, 0xc42015aba0, 0x6)
    /root/workspace/zagen/src/ +0x1f9*Session).innerInsert(0xc42012b180, 0x9628c0, 0xc42038d860, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/ +0xf90*Session).Insert(0xc42012b180, 0xc420049a68, 0x1, 0x1, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/ +0x327*Engine).Insert(0xc420255720, 0xc420049a68, 0x1, 0x1, 0x0, 0x0, 0x0)
    /root/workspace/zagen/src/ +0xa4
zagen/fw.(*DBO).insertKeepOld(0xc4202548c0, 0xe3f360, 0xc42038d860, 0xe3f360, 0xc42038dbc0, 0x0, 0x9a84e0, 0xc4204b4801, 0xc42038dbc0)
    /root/workspace/zagen/src/zagen/fw/dbo.go:307 +0x517
zagen/fw.(*DBO).Insert(0xc4202548c0, 0xe3f360, 0xc42038d860, 0xe3f360, 0xc42038dbc0, 0xc4201fa000, 0x80, 0x100)
    /root/workspace/zagen/src/zagen/fw/dbo.go:254 +0x58
zplug/pkg.Collect(0xe00000000, 0x0, 0x0)
    /root/workspace/zagen/src/zplug/pkg/main.go:118 +0xfa4
    /root/workspace/zagen/src/zagen/core/proc.go:136 +0x180
created by zagen/core.(*procRuntime).scheduleRoutine
    /root/workspace/zagen/src/zagen/core/proc.go:208 +0x1b8

goroutine 37 [GC assist wait]:
regexp.(*bitState).reset(0xc420184300, 0x5f, 0x2)
    /usr/local/go/src/regexp/backtrack.go:95 +0xff
regexp.(*machine).backtrack(0xc42048b680, 0xe41e80, 0xc42048b758, 0x0, 0x5f, 0x2, 0x7)
    /usr/local/go/src/regexp/backtrack.go:321 +0xa1
regexp.(*Regexp).doExecute(0xc42010cbe0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4201841e0, 0x5f, 0x0, 0x2, ...)
    /usr/local/go/src/regexp/exec.go:437 +0x2e5
regexp.(*Regexp).replaceAll(0xc42010cbe0, 0x0, 0x0, 0x0, 0xc4201841e0, 0x5f, 0x2, 0xc420395850, 0xc4201841e0, 0x5f, ...)
    /usr/local/go/src/regexp/regexp.go:506 +0x10c
regexp.(*Regexp).ReplaceAllString(0xc42010cbe0, 0xc4201841e0, 0x5f, 0x9dbd48, 0x1, 0x5f, 0x6)
    /usr/local/go/src/regexp/regexp.go:467 +0x125
zplug/process.getProcModule(0x3e5, 0xc42046d220, 0x1, 0x1)
    /root/workspace/zagen/src/zplug/process/proc.go:58 +0x1f4
zplug/process.GetProcDetail(0x3e5, 0x3)
    /root/workspace/zagen/src/zplug/process/proc.go:253 +0x5fa
zplug/process.collect(0x0, 0x0)
    /root/workspace/zagen/src/zplug/process/collect.go:47 +0x432
zplug/process.Main(0xe00000030, 0xa2e030, 0xc42019678c)
    /root/workspace/zagen/src/zplug/process/main.go:4 +0x22
    /root/workspace/zagen/src/zagen/core/proc.go:136 +0x180
created by zagen/core.(*procRuntime).scheduleRoutine
    /root/workspace/zagen/src/zagen/core/proc.go:208 +0x1b8

goroutine 149 [chan receive, 1 minutes]:
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 92 [chan receive, 1 minutes]:
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 61 [chan receive, 1 minutes]:
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 1251 [select]:*SQLiteStmt).exec.func1(0x7f2d815061e8, 0xc4200122c8, 0xc42015b020, 0x21c61a8)
    /root/workspace/zagen/src/ +0x17b
created by*SQLiteStmt).exec
    /root/workspace/zagen/src/ +0x285

goroutine 452 [chan receive, 1 minutes]:
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 154 [chan receive, 1 minutes]:
    /usr/local/go/src/database/sql/sql.go:730 +0x4a
created by database/sql.Open
    /usr/local/go/src/database/sql/sql.go:493 +0x1e9

goroutine 1253 [runnable]:*SQLiteStmt).exec.func1(0x7f2d815061e8, 0xc4200122c8, 0xc42015b200, 0x7f2d54014dd8)
    /root/workspace/zagen/src/ +0x17b
created by*SQLiteStmt).exec
    /root/workspace/zagen/src/ +0x285

This comment has been minimized.

Copy link

commented Apr 13, 2017

This might've already been fixed in Go 1.8. Could you try Go 1.8.1?


This comment has been minimized.

Copy link

commented Apr 15, 2017

@t57root does this appear in Go 1.8?


This comment has been minimized.

Copy link

commented Apr 16, 2017

Sorry about the late reply. According to my experience so far, seems it doesn't appear in Go 1.8. because it's not inevitable.
if this have been fixed in go 1.8, will the fix be backported to 1.7?


This comment has been minimized.

Copy link

commented Apr 16, 2017

We generally only backport security issues. We've occasionally backported runtime crashes, so this might qualify, but it's not clear it's affecting enough people to be worth it. We'd encourage you to use Go 1.8 instead.

/cc @aclements for his opinion.


This comment has been minimized.

Copy link

commented Apr 17, 2017

I agree with @bradfitz that this doesn't seem to qualify for a backport. Lots of things were fixed in 1.8, so we'd much rather users move forward (or tell us if there's an issue in Go preventing them for doing so!). Closing for now, but if you disagree or discover that you can reproduce it with 1.8, please comment and we'll reopen.

@aclements aclements closed this Apr 17, 2017


This comment has been minimized.

Copy link

commented Nov 9, 2017

Hi, @bradfitz @aclements I am using Go 1.9.2 and I got the same error as OP after several billion runs of a Go program. I happen to have the core dump, which I can send you in private if you wish to analyze it further. Please let me know, thanks.


This comment has been minimized.

Copy link

commented Nov 9, 2017

@guidovranken Can you show us the stack trace? I'd like to see if this looks like the same issue or a different one.

Note that one error after several billion runs could easily be a random memory failure. Even if the bug is in Go if it takes several billion runs to repeat the problem we are unlikely to be able to fix it.


This comment has been minimized.

Copy link

commented Nov 12, 2017

@ianlancetaylor due to reasons I only have this screenshot of the original crash:

In case it is relevant, the Go program was compiled as a static library and launched from a C++ program (a fuzzer actually, hence the billions of runs).

@golang golang locked and limited conversation to collaborators Nov 12, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.