Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: goroutine hung in _Cfunc_.GoString #30098

Closed
cliffburdick opened this issue Feb 5, 2019 · 14 comments

Comments

Projects
None yet
8 participants
@cliffburdick
Copy link

commented Feb 5, 2019

$ go version
go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/cburdick/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/cburdick/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/cburdick/myapp/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build475396611=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I have an application that uses cgo to link to DPDK. After running for about 10 minutes, the main goroutine hangs. When I look in dlv at what it's doing, it's stuck in _Cfunc_GoString with the backtrace:

0 0x00000000004d1230 in runtime.systemstack_switch
at .usr/local/go/src/runtime/asm_amd64.s:311
1 0x0000000000495b4f in runtime.gcStart
at .usr/local/go/src/runtime/mgc.go:1316
2 0x0000000000487dfe in runtime.mallocgc
at .usr/local/go/src/runtime/malloc.go:996
3 0x00000000004c0c9f in runtime.rawstring
at .usr/local/go/src/runtime/string.go:258
4 0x00000000004c1047 in runtime.gostring
at .usr/local/go/src/runtime/string.go:317
5 0x0000000000825afb in github.com/cburdick/myapp/internal/dpdk._Cfunc_GoString
at _cgo_gotypes.go:869
6 0x0000000000827bcc in github.com/cburdick/myapp/internal/dpdk.(*Dpdk).DpdkMain
at .home/cburdick/myapp/internal/dpdk/dpdk.go:178
7 0x000000000083433c in github.com/cburdick/myapp/cmd/myapp/app.Run.func1
at .home/cburdick/myapp/cmd/myapp/app/myapp.go:32
8 0x00000000004d3311 in runtime.goexit
at .usr/local/go/src/runtime/asm_amd64.s:1333

The actual line where it hangs is:
name := C.GoString((*C.char)(unsafe.Pointer(&msg.name[0])))

Where msg.name is a NULL-terminated C character array.

What did you expect to see?

No hanging

@odeke-em odeke-em changed the title Goroutine Hung in _Cfunc_GoString runtime: goroutine hung in _Cfunc_.GoString Feb 6, 2019

@odeke-em

This comment has been minimized.

Copy link
Member

commented Feb 6, 2019

Thank you for the bug report and welcome to the Go project @cliffburdick!

I'll kindly page some experts @randall77 @ianlancetaylor @aclements.

@aclements

This comment has been minimized.

Copy link
Member

commented Feb 6, 2019

Thanks for the report. Seeing more of the stack trace will help us debug this. Can you set GOTRACEBACK=crash in the environment, reproduce the issue, and then send a SIGQUIT to the hung process (either using kill or by pressing Ctrl-)?

@cliffburdick

This comment has been minimized.

Copy link
Author

commented Feb 6, 2019

Hi @aclements, here is the output after a SIGQUIT. I did notice that it's now hanging in a different section that's printing a C.uint16:

SIGQUIT: quit
PC=0xa32fda m=0 sigcode=0

goroutine 1 [running]:
github.com/cburdick/myapp/cmd/myapp/app.Run(0xc000185f78, 0x1)
        /home/cburdick/myapp/cmd/myapp/app/myapp.go:25 +0x9a fp=0xc000185f40 sp=0xc000185ed0 pc=0xa32fda
main.main()
        /home/cburdick/myapp/cmd/myapp/main.go:10 +0x74 fp=0xc000185f98 sp=0xc000185f40 pc=0xa333d4
runtime.main()
        /usr/local/go/src/runtime/proc.go:201 +0x207 fp=0xc000185fe0 sp=0xc000185f98 pc=0x4b6da7
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000185fe8 sp=0xc000185fe0 pc=0x4e3291

goroutine 2 [force gc (idle)]:
runtime.gopark(0x10be2e8, 0x18ea8d0, 0x2cd1410, 0x1)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000faf80 sp=0xc0000faf60 pc=0x4b71bb
runtime.goparkunlock(0x18ea8d0, 0x1410, 0x1)
        /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc0000fafb0 sp=0xc0000faf80 pc=0x4b7263
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:251 +0xb3 fp=0xc0000fafe0 sp=0xc0000fafb0 pc=0x4b7033
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000fafe8 sp=0xc0000fafe0 pc=0x4e3291
created by runtime.init.4
        /usr/local/go/src/runtime/proc.go:240 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x10be2e8, 0x18eb180, 0x4a140c, 0x1)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000fb780 sp=0xc0000fb760 pc=0x4b71bb
runtime.goparkunlock(0x18eb180, 0x113140c, 0x1)
        /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc0000fb7b0 sp=0xc0000fb780 pc=0x4b7263
runtime.bgsweep(0xc000122000)
        /usr/local/go/src/runtime/mgcsweep.go:52 +0x8f fp=0xc0000fb7d8 sp=0xc0000fb7b0 pc=0x4aa24f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000fb7e0 sp=0xc0000fb7d8 pc=0x4e3291
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:216 +0x58

goroutine 4 [finalizer wait]:
runtime.gopark(0x10be2e8, 0x27745b0, 0x2cc140f, 0x1)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000fbf28 sp=0xc0000fbf08 pc=0x4b71bb
runtime.goparkunlock(0x27745b0, 0x140f, 0x1)
        /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc0000fbf58 sp=0xc0000fbf28 pc=0x4b7263
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:175 +0x99 fp=0xc0000fbfe0 sp=0xc0000fbf58 pc=0x4a1c29
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0x4e3291
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:156 +0x61

goroutine 5 [running]:
        goroutine running on other thread; stack unavailable
created by github.com/cburdick/myapp/cmd/myapp/app.Run
        /home/cburdick/myapp/cmd/myapp/app/myapp.go:25 +0x9a

goroutine 18 [timer goroutine (idle)]:
runtime.gopark(0x10be2e8, 0x18ef8a0, 0x1414, 0x1)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000fa728 sp=0xc0000fa708 pc=0x4b71bb
runtime.goparkunlock(0x18ef8a0, 0xc000001414, 0x1)
        /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc0000fa758 sp=0xc0000fa728 pc=0x4b7263
runtime.timerproc(0x18ef8a0)
        /usr/local/go/src/runtime/time.go:280 +0x296 fp=0xc0000fa7d8 sp=0xc0000fa758 pc=0x4d48c6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000fa7e0 sp=0xc0000fa7d8 pc=0x4e3291
created by runtime.(*timersBucket).addtimerLocked
        /usr/local/go/src/runtime/time.go:170 +0x114

goroutine 34 [timer goroutine (idle)]:
runtime.gopark(0x10be2e8, 0x18ef920, 0x1414, 0x1)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f6728 sp=0xc0000f6708 pc=0x4b71bb
runtime.goparkunlock(0x18ef920, 0xc000001414, 0x1)
        /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc0000f6758 sp=0xc0000f6728 pc=0x4b7263
runtime.timerproc(0x18ef920)
        /usr/local/go/src/runtime/time.go:280 +0x296 fp=0xc0000f67d8 sp=0xc0000f6758 pc=0x4d48c6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f67e0 sp=0xc0000f67d8 pc=0x4e3291
created by runtime.(*timersBucket).addtimerLocked
        /usr/local/go/src/runtime/time.go:170 +0x114

goroutine 50 [timer goroutine (idle)]:
runtime.gopark(0x10be2e8, 0x18ef9a0, 0x1414, 0x1)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002d8728 sp=0xc0002d8708 pc=0x4b71bb
runtime.goparkunlock(0x18ef9a0, 0xc000001414, 0x1)
        /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc0002d8758 sp=0xc0002d8728 pc=0x4b7263
runtime.timerproc(0x18ef9a0)
        /usr/local/go/src/runtime/time.go:280 +0x296 fp=0xc0002d87d8 sp=0xc0002d8758 pc=0x4d48c6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002d87e0 sp=0xc0002d87d8 pc=0x4e3291
created by runtime.(*timersBucket).addtimerLocked
        /usr/local/go/src/runtime/time.go:170 +0x114

goroutine 56 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c10, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002da760 sp=0xc0002da740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00009b900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0002da7d8 sp=0xc0002da760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002da7e0 sp=0xc0002da7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 55 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a7b0, 0xc0002d1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002d9760 sp=0xc0002d9740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000099400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0002d97d8 sp=0xc0002d9760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002d97e0 sp=0xc0002d97d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 66 [timer goroutine (idle)]:
runtime.gopark(0x10be2e8, 0x18efa20, 0x1414, 0x1)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002d4728 sp=0xc0002d4708 pc=0x4b71bb
runtime.goparkunlock(0x18efa20, 0xc000001414, 0x1)
        /usr/local/go/src/runtime/proc.go:308 +0x53 fp=0xc0002d4758 sp=0xc0002d4728 pc=0x4b7263
runtime.timerproc(0x18efa20)
        /usr/local/go/src/runtime/time.go:280 +0x296 fp=0xc0002d47d8 sp=0xc0002d4758 pc=0x4d48c6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002d47e0 sp=0xc0002d47d8 pc=0x4e3291
created by runtime.(*timersBucket).addtimerLocked
        /usr/local/go/src/runtime/time.go:170 +0x114

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415b60, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f6f60 sp=0xc0000f6f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000032a00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0000f6fd8 sp=0xc0000f6f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f6fe0 sp=0xc0000f6fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a730, 0xc000351417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035a760 sp=0xc00035a740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000030500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035a7d8 sp=0xc00035a760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035a7e0 sp=0xc00035a7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 54 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c00, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002d9f60 sp=0xc0002d9f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000094a00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0002d9fd8 sp=0xc0002d9f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002d9fe0 sp=0xc0002d9fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 53 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415bf0, 0xc0000f1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f7760 sp=0xc0000f7740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000090000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0000f77d8 sp=0xc0000f7760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f77e0 sp=0xc0000f77d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a720, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f7f60 sp=0xc0000f7f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00002e000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0000f7fd8 sp=0xc0000f7f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f7fe0 sp=0xc0000f7fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a740, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035af60 sp=0xc00035af40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000034f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035afd8 sp=0xc00035af60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035afe0 sp=0xc00035afd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 24 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d280, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035b760 sp=0xc00035b740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000037400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035b7d8 sp=0xc00035b760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035b7e0 sp=0xc00035b7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 25 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d290, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035bf60 sp=0xc00035bf40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000039900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035bfd8 sp=0xc00035bf60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035bfe0 sp=0xc00035bfd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 26 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415b70, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035c760 sp=0xc00035c740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00003c000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035c7d8 sp=0xc00035c760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035c7e0 sp=0xc00035c7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 27 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a750, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035cf60 sp=0xc00035cf40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00003e500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035cfd8 sp=0xc00035cf60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035cfe0 sp=0xc00035cfd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 28 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415b80, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035d760 sp=0xc00035d740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000040a00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035d7d8 sp=0xc00035d760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035d7e0 sp=0xc00035d7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 29 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d2a0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00035df60 sp=0xc00035df40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000042f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00035dfd8 sp=0xc00035df60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00035dfe0 sp=0xc00035dfd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 30 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d2b0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000356760 sp=0xc000356740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000045400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0003567d8 sp=0xc000356760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003567e0 sp=0xc0003567d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 31 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d2c0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000356f60 sp=0xc000356f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000047900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000356fd8 sp=0xc000356f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000356fe0 sp=0xc000356fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 32 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d2d0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000357760 sp=0xc000357740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00004a000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0003577d8 sp=0xc000357760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003577e0 sp=0xc0003577d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d2e0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000357f60 sp=0xc000357f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00004c500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000357fd8 sp=0xc000357f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000357fe0 sp=0xc000357fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 82 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d2f0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000358760 sp=0xc000358740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00004ea00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0003587d8 sp=0xc000358760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003587e0 sp=0xc0003587d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 83 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d300, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000358f60 sp=0xc000358f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000050f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000358fd8 sp=0xc000358f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000358fe0 sp=0xc000358fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 84 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d310, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000359760 sp=0xc000359740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000053400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0003597d8 sp=0xc000359760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0003597e0 sp=0xc0003597d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 85 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d320, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000359f60 sp=0xc000359f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000055900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000359fd8 sp=0xc000359f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000359fe0 sp=0xc000359fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 86 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d330, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000486760 sp=0xc000486740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000058000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004867d8 sp=0xc000486760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004867e0 sp=0xc0004867d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 87 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a760, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000486f60 sp=0xc000486f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00005a500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000486fd8 sp=0xc000486f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000486fe0 sp=0xc000486fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 88 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d340, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000487760 sp=0xc000487740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00005ca00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004877d8 sp=0xc000487760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004877e0 sp=0xc0004877d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 89 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d350, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000487f60 sp=0xc000487f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00005ef00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000487fd8 sp=0xc000487f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000487fe0 sp=0xc000487fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 90 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d360, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000488760 sp=0xc000488740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000061400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004887d8 sp=0xc000488760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004887e0 sp=0xc0004887d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 91 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d370, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000488f60 sp=0xc000488f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000063900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000488fd8 sp=0xc000488f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000488fe0 sp=0xc000488fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 92 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415b90, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000489760 sp=0xc000489740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000066000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004897d8 sp=0xc000489760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004897e0 sp=0xc0004897d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 93 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a770, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000489f60 sp=0xc000489f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000068500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000489fd8 sp=0xc000489f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000489fe0 sp=0xc000489fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 94 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d380, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000482760 sp=0xc000482740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00006aa00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004827d8 sp=0xc000482760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004827e0 sp=0xc0004827d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 95 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415ba0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000482f60 sp=0xc000482f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00006cf00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000482fd8 sp=0xc000482f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000482fe0 sp=0xc000482fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 96 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415bb0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000483760 sp=0xc000483740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00006f400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004837d8 sp=0xc000483760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004837e0 sp=0xc0004837d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 97 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d390, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000483f60 sp=0xc000483f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000071900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000483fd8 sp=0xc000483f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000483fe0 sp=0xc000483fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a780, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f8760 sp=0xc0000f8740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000074000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0000f87d8 sp=0xc0000f8760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f87e0 sp=0xc0000f87d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 98 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415bc0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000484760 sp=0xc000484740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000076500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004847d8 sp=0xc000484760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004847e0 sp=0xc0004847d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 99 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d3a0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000484f60 sp=0xc000484f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000078a00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000484fd8 sp=0xc000484f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000484fe0 sp=0xc000484fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 100 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a790, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000485760 sp=0xc000485740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00007af00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004857d8 sp=0xc000485760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004857e0 sp=0xc0004857d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 101 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d3b0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000485f60 sp=0xc000485f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00007d400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000485fd8 sp=0xc000485f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000485fe0 sp=0xc000485fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 102 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415bd0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000490760 sp=0xc000490740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00007f900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004907d8 sp=0xc000490760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004907e0 sp=0xc0004907d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 103 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d3c0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000490f60 sp=0xc000490f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000082000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000490fd8 sp=0xc000490f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000490fe0 sp=0xc000490fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 104 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a7a0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000491760 sp=0xc000491740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000084500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004917d8 sp=0xc000491760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004917e0 sp=0xc0004917d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 105 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d3d0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000491f60 sp=0xc000491f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000086a00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000491fd8 sp=0xc000491f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000491fe0 sp=0xc000491fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 106 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415be0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000492760 sp=0xc000492740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000088f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004927d8 sp=0xc000492760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004927e0 sp=0xc0004927d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 107 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d3e0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000492f60 sp=0xc000492f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00008b400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000492fd8 sp=0xc000492f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000492fe0 sp=0xc000492fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 108 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d3f0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000493760 sp=0xc000493740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00008d900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004937d8 sp=0xc000493760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004937e0 sp=0xc0004937d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 109 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d400, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000493f60 sp=0xc000493f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000092500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000493fd8 sp=0xc000493f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000493fe0 sp=0xc000493fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 110 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d410, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048c760 sp=0xc00048c740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc000096f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048c7d8 sp=0xc00048c760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048c7e0 sp=0xc00048c7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 111 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a7c0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048cf60 sp=0xc00048cf40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc00009e000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048cfd8 sp=0xc00048cf60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048cfe0 sp=0xc00048cfd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 112 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d420, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048d760 sp=0xc00048d740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000a0500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048d7d8 sp=0xc00048d760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048d7e0 sp=0xc00048d7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 113 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d430, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048df60 sp=0xc00048df40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000a2a00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048dfd8 sp=0xc00048df60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048dfe0 sp=0xc00048dfd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 57 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c20, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002daf60 sp=0xc0002daf40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000a4f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0002dafd8 sp=0xc0002daf60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002dafe0 sp=0xc0002dafd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 114 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d440, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048e760 sp=0xc00048e740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000a7400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048e7d8 sp=0xc00048e760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048e7e0 sp=0xc00048e7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 58 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c30, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002db760 sp=0xc0002db740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000a9900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0002db7d8 sp=0xc0002db760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002db7e0 sp=0xc0002db7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 115 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d450, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048ef60 sp=0xc00048ef40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000ac000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048efd8 sp=0xc00048ef60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048efe0 sp=0xc00048efd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 59 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a7d0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0002dbf60 sp=0xc0002dbf40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000ae500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0002dbfd8 sp=0xc0002dbf60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0002dbfe0 sp=0xc0002dbfd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 60 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c40, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00049a760 sp=0xc00049a740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000b0a00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00049a7d8 sp=0xc00049a760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00049a7e0 sp=0xc00049a7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 116 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a7e0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048f760 sp=0xc00048f740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000b2f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048f7d8 sp=0xc00048f760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048f7e0 sp=0xc00048f7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 117 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d460, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00048ff60 sp=0xc00048ff40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000b5400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00048ffd8 sp=0xc00048ff60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00048ffe0 sp=0xc00048ffd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 118 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c50, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000496760 sp=0xc000496740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000b7900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004967d8 sp=0xc000496760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004967e0 sp=0xc0004967d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 119 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d470, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000496f60 sp=0xc000496f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000ba000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000496fd8 sp=0xc000496f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000496fe0 sp=0xc000496fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a7f0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f8f60 sp=0xc0000f8f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000bc500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0000f8fd8 sp=0xc0000f8f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f8fe0 sp=0xc0000f8fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 120 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d480, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000497760 sp=0xc000497740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000bea00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004977d8 sp=0xc000497760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004977e0 sp=0xc0004977d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a800, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f9760 sp=0xc0000f9740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000c0f00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0000f97d8 sp=0xc0000f9760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f97e0 sp=0xc0000f97d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 121 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c60, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000497f60 sp=0xc000497f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000c3400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000497fd8 sp=0xc000497f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000497fe0 sp=0xc000497fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 122 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d490, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000498760 sp=0xc000498740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000c5900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004987d8 sp=0xc000498760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004987e0 sp=0xc0004987d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 123 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d4a0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000498f60 sp=0xc000498f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000c8000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000498fd8 sp=0xc000498f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000498fe0 sp=0xc000498fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 61 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c70, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00049af60 sp=0xc00049af40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000ca500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00049afd8 sp=0xc00049af60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00049afe0 sp=0xc00049afd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 124 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a810, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000499760 sp=0xc000499740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000cca00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004997d8 sp=0xc000499760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004997e0 sp=0xc0004997d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 125 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d4b0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc000499f60 sp=0xc000499f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000cef00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc000499fd8 sp=0xc000499f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000499fe0 sp=0xc000499fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 126 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c80, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0004a2760 sp=0xc0004a2740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000d1400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004a27d8 sp=0xc0004a2760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004a27e0 sp=0xc0004a27d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 127 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d4c0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0004a2f60 sp=0xc0004a2f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000d3900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004a2fd8 sp=0xc0004a2f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004a2fe0 sp=0xc0004a2fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 41 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a820, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0000f9f60 sp=0xc0000f9f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000d6000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0000f9fd8 sp=0xc0000f9f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0000f9fe0 sp=0xc0000f9fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 128 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415c90, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0004a3760 sp=0xc0004a3740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000d8500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004a37d8 sp=0xc0004a3760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004a37e0 sp=0xc0004a37d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 129 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d4d0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0004a3f60 sp=0xc0004a3f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000daa00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004a3fd8 sp=0xc0004a3f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004a3fe0 sp=0xc0004a3fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 130 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046d4e0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0004a4760 sp=0xc0004a4740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000dcf00)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004a47d8 sp=0xc0004a4760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004a47e0 sp=0xc0004a47d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 62 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a830, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00049b760 sp=0xc00049b740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000df400)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00049b7d8 sp=0xc00049b760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00049b7e0 sp=0xc00049b7d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 63 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415ca0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc00049bf60 sp=0xc00049bf40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000e1900)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc00049bfd8 sp=0xc00049bf60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc00049bfe0 sp=0xc00049bfd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 131 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc00046a840, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0004a4f60 sp=0xc0004a4f40 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000e4000)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004a4fd8 sp=0xc0004a4f60 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004a4fe0 sp=0xc0004a4fd8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

goroutine 132 [GC worker (idle)]:
runtime.gopark(0x10be100, 0xc000415cb0, 0x1417, 0x0)
        /usr/local/go/src/runtime/proc.go:302 +0xeb fp=0xc0004a5760 sp=0xc0004a5740 pc=0x4b71bb
runtime.gcBgMarkWorker(0xc0000e6500)
        /usr/local/go/src/runtime/mgc.go:1772 +0xfc fp=0xc0004a57d8 sp=0xc0004a5760 pc=0x4a54fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0004a57e0 sp=0xc0004a57d8 pc=0x4e3291
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1720 +0x77

rax    0xc000000300
rbx    0x18ec300
rcx    0x18ebe31
rdx    0x3
rdi    0xe2efb22e
rsi    0x3
rbp    0xc000185f30
rsp    0xc000185ed0
r8     0x2cf3440
r9     0x0
r10    0x2c5c010
r11    0x0
r12    0x7fff90c26d50
r13    0x18ec300
r14    0x18ebd80
r15    0x0
rip    0xa32fda
rflags 0x3202
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x4e55b3 m=2 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x18ebbb0, 0x80, 0x7f834ed8b590, 0x0, 0x7f8300000000, 0x4e51e6, 0x3c, 0x0, 0x7f834ed8b5a8, 0x49495e, ...)
        /usr/local/go/src/runtime/sys_linux_amd64.s:532 +0x23 fp=0x7f834ed8b560 sp=0x7f834ed8b558 pc=0x4e55b3
runtime.futexsleep(0x18ebbb0, 0x7f8300000000, 0xdf8475800)
        /usr/local/go/src/runtime/os_linux.go:63 +0xc0 fp=0x7f834ed8b5b0 sp=0x7f834ed8b560 pc=0x4b2f70
runtime.notetsleep_internal(0x18ebbb0, 0xdf8475800, 0x989680)
        /usr/local/go/src/runtime/lock_futex.go:193 +0x12e fp=0x7f834ed8b5e8 sp=0x7f834ed8b5b0 pc=0x49495e
runtime.notetsleep(0x18ebbb0, 0xdf8475800, 0x0)
        /usr/local/go/src/runtime/lock_futex.go:216 +0x54 fp=0x7f834ed8b610 sp=0x7f834ed8b5e8 pc=0x494a34
runtime.sysmon()
        /usr/local/go/src/runtime/proc.go:4369 +0x173 fp=0x7f834ed8b698 sp=0x7f834ed8b610 pc=0x4c1773
runtime.mstart1()
        /usr/local/go/src/runtime/proc.go:1282 +0xe6 fp=0x7f834ed8b6c0 sp=0x7f834ed8b698 pc=0x4b9e96
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1248 +0x6e fp=0x7f834ed8b6e8 sp=0x7f834ed8b6c0 pc=0x4b9d8e
rax    0xfffffffffffffffc
rbx    0x0
rcx    0x4e55b3
rdx    0x0
rdi    0x18ebbb0
rsi    0x80
rbp    0x7f834ed8b5a0
rsp    0x7f834ed8b558
r8     0x0
r9     0x0
r10    0x7f834ed8b590
r11    0x3246
r12    0x7f834ed8b800
r13    0x0
r14    0x2c62ca0
r15    0x7fff90c26c10
rip    0x4e55b3
rflags 0x3246
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x4e5760 m=3 sigcode=0

goroutine 0 [idle]:
runtime.epollwait(0x7f8300000004, 0x7f834e589f90, 0xffffffff00000080, 0x7f8300000000, 0xffffffff4e58a21a, 0x100000000, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/local/go/src/runtime/sys_linux_amd64.s:671 +0x20 fp=0x7f834e589f58 sp=0x7f834e589f50 pc=0x4e5760
runtime.netpoll(0xc0000e6501, 0xc000078a01)
        /usr/local/go/src/runtime/netpoll_epoll.go:71 +0x132 fp=0x7f834e58a5a0 sp=0x7f834e589f58 pc=0x4b2db2
runtime.findrunnable(0xc000032a00, 0x0)
        /usr/local/go/src/runtime/proc.go:2469 +0x51c fp=0x7f834e58a640 sp=0x7f834e58a5a0 pc=0x4bc50c
runtime.schedule()
        /usr/local/go/src/runtime/proc.go:2613 +0x13a fp=0x7f834e58a688 sp=0x7f834e58a640 pc=0x4bcfda
runtime.park_m(0xc000001980)
        /usr/local/go/src/runtime/proc.go:2676 +0xae fp=0x7f834e58a6b8 sp=0x7f834e58a688 pc=0x4bd2fe
runtime.mcall(0x800000)
        /usr/local/go/src/runtime/asm_amd64.s:299 +0x5b fp=0x7f834e58a6c8 sp=0x7f834e58a6b8 pc=0x4e119b
rax    0xfffffffffffffffc
rbx    0xffffffff
rcx    0x4e5760
rdx    0x80
rdi    0x4
rsi    0x7f834e589f90
rbp    0x7f834e58a590
rsp    0x7f834e589f50
r8     0x0
r9     0x3
r10    0xffffffff
r11    0x3202
r12    0xc0002e0660
r13    0xc0000fe700
r14    0xc000001980
r15    0x0
rip    0x4e5760
rflags 0x3202
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x4e55b1 m=4 sigcode=0

goroutine 0 [idle]:
runtime.futex(0xc0000febc0, 0x80, 0x0, 0x0, 0x0, 0xc0004a64b0, 0x0, 0x0, 0x7f834dd89568, 0x4947d2, ...)
        /usr/local/go/src/runtime/sys_linux_amd64.s:531 +0x21 fp=0x7f834dd894f0 sp=0x7f834dd894e8 pc=0x4e55b1
runtime.futexsleep(0xc0000febc0, 0x0, 0xffffffffffffffff)
        /usr/local/go/src/runtime/os_linux.go:46 +0x4b fp=0x7f834dd89540 sp=0x7f834dd894f0 pc=0x4b2efb
runtime.notesleep(0xc0000febc0)
        /usr/local/go/src/runtime/lock_futex.go:151 +0xa2 fp=0x7f834dd89578 sp=0x7f834dd89540 pc=0x4947d2
runtime.stopm()
        /usr/local/go/src/runtime/proc.go:2016 +0xe3 fp=0x7f834dd895a0 sp=0x7f834dd89578 pc=0x4bb353
runtime.findrunnable(0xc000034f00, 0x0)
        /usr/local/go/src/runtime/proc.go:2487 +0x4dc fp=0x7f834dd89640 sp=0x7f834dd895a0 pc=0x4bc4cc
runtime.schedule()
        /usr/local/go/src/runtime/proc.go:2613 +0x13a fp=0x7f834dd89688 sp=0x7f834dd89640 pc=0x4bcfda
runtime.park_m(0xc0004a6480)
        /usr/local/go/src/runtime/proc.go:2676 +0xae fp=0x7f834dd896b8 sp=0x7f834dd89688 pc=0x4bd2fe
runtime.mcall(0x800000)
        /usr/local/go/src/runtime/asm_amd64.s:299 +0x5b fp=0x7f834dd896c8 sp=0x7f834dd896b8 pc=0x4e119b
rax    0xca
rbx    0xc0000fea80
rcx    0x4e55b3
rdx    0x0
rdi    0xc0000febc0
rsi    0x80
rbp    0x7f834dd89530
rsp    0x7f834dd894e8
r8     0x0
r9     0x0
r10    0x0
r11    0x3286
r12    0xc0004a5648
r13    0xc0000fea80
r14    0xc0004a6480
r15    0x200
rip    0x4e55b1
rflags 0x3286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x4e55b1 m=5 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x2774978, 0x80, 0x0, 0x0, 0x0, 0x7f834d588658, 0x0, 0x0, 0x7f834d588668, 0x4947d2, ...)
        /usr/local/go/src/runtime/sys_linux_amd64.s:531 +0x21 fp=0x7f834d5885f0 sp=0x7f834d5885e8 pc=0x4e55b1
runtime.futexsleep(0x2774978, 0x7f8300000000, 0xffffffffffffffff)
        /usr/local/go/src/runtime/os_linux.go:46 +0x4b fp=0x7f834d588640 sp=0x7f834d5885f0 pc=0x4b2efb
runtime.notesleep(0x2774978)
        /usr/local/go/src/runtime/lock_futex.go:151 +0xa2 fp=0x7f834d588678 sp=0x7f834d588640 pc=0x4947d2
runtime.templateThread()
        /usr/local/go/src/runtime/proc.go:1993 +0xf9 fp=0x7f834d588698 sp=0x7f834d588678 pc=0x4bb249
runtime.mstart1()
        /usr/local/go/src/runtime/proc.go:1282 +0xe6 fp=0x7f834d5886c0 sp=0x7f834d588698 pc=0x4b9e96
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1248 +0x6e fp=0x7f834d5886e8 sp=0x7f834d5886c0 pc=0x4b9d8e
rax    0xca
rbx    0xc0000fee00
rcx    0x4e55b3
rdx    0x0
rdi    0x2774978
rsi    0x80
rbp    0x7f834d588630
rsp    0x7f834d5885e8
r8     0x0
r9     0x0
r10    0x0
r11    0x3286
r12    0x7f834d588800
r13    0x0
r14    0x2cd3120
r15    0x7fff90c26cc0
rip    0x4e55b1
rflags 0x3286
cs     0x33
fs     0x0
gs     0x0

-----

Quit (core dumped)

Here was the dlv bt:

(dlv) goroutine 5
Switched from 0 to 5 (thread 675)
(dlv) bt
 0  0x00000000004e11b0 in runtime.systemstack_switch
    at .usr/local/go/src/runtime/asm_amd64.s:311
 1  0x00000000004a41df in runtime.gcStart
    at .usr/local/go/src/runtime/mgc.go:1316
 2  0x00000000004959e9 in runtime.mallocgc
    at .usr/local/go/src/runtime/malloc.go:996
 3  0x0000000000496158 in runtime.newobject
    at .usr/local/go/src/runtime/malloc.go:1032
 4  0x00000000005f083b in github.com/sirupsen/logrus.init.0.func1
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:17
 5  0x00000000005027a8 in sync.(*Pool).Get
    at .usr/local/go/src/sync/pool.go:151
 6  0x00000000005e737c in github.com/sirupsen/logrus.Entry.log
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:125
 7  0x00000000005e7ef6 in github.com/sirupsen/logrus.(*Entry).Info
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:177
 8  0x00000000005e8854 in github.com/sirupsen/logrus.(*Entry).Infof
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:221
 9  0x00000000005eb351 in github.com/sirupsen/logrus.(*Logger).Infof
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/logger.go:135
10  0x00000000005e998d in github.com/sirupsen/logrus.Infof
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/exported.go:132
11  0x0000000000a1fe41 in github.com/cburdick/myapp/internal/dpdk.(*Dpdk).DpdkMain
    at .home/cburdick/myapp/internal/dpdk/dpdk.go:177
12  0x0000000000a33252 in github.com/cburdick/myapp/cmd/myapp/app.Run.func1
    at .home/cburdick/myapp/cmd/myapp/app/myapp.go:32
13  0x00000000004e3291 in runtime.goexit
    at .usr/local/go/src/runtime/asm_amd64.s:1333
(dlv) threads
* Thread 675 at 0xa32fda .home/cburdick/myapp/cmd/myapp/app/myapp.go:25 github.com/cburdick/myapp/cmd/myapp/app.Run
  Thread 676 at 0x4e55b3 .usr/local/go/src/runtime/sys_linux_amd64.s:532 runtime.futex
  Thread 677 at 0x4e55b3 .usr/local/go/src/runtime/sys_linux_amd64.s:532 runtime.futex
  Thread 678 at 0x4e55b3 .usr/local/go/src/runtime/sys_linux_amd64.s:532 runtime.futex
  Thread 679 at 0x4e55b3 .usr/local/go/src/runtime/sys_linux_amd64.s:532 runtime.futex
  Thread 680 at :0
  Thread 681 at :0
  Thread 682 at :0
  Thread 683 at 0x4e5760 .usr/local/go/src/runtime/sys_linux_amd64.s:671 runtime.epollwait
  Thread 685 at 0x4e55b3 .usr/local/go/src/runtime/sys_linux_amd64.s:532 runtime.futex
  Thread 706 at 0x4e55b3 .usr/local/go/src/runtime/sys_linux_amd64.s:532 runtime.futex

EDITED to add the <details> to reduce on scrolling from huge stack report.
EDITED to use fixed-width formatting.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2019

I suspect that you have a non-preemptible loop. See #10958. In particular, can you use the "created by" line to figure out what code this goroutine is running?

goroutine 5 [running]:
        goroutine running on other thread; stack unavailable
created by github.com/cburdick/myapp/cmd/myapp/app.Run
        /home/cburdick/myapp/cmd/myapp/app/myapp.go:25 +0x9a
@cliffburdick

This comment has been minimized.

Copy link
Author

commented Feb 6, 2019

@ianlancetaylor Is the dlv output I posted in the last message what you were looking for? It has a stack trace that was caused by logrus.

@andybons andybons added this to the Unplanned milestone Feb 6, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2019

Thanks, you're right. I don't know what is going on here.

@ianlancetaylor ianlancetaylor modified the milestones: Unplanned, Go1.13 Feb 6, 2019

@randall77

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2019

What is your code doing here:

/home/cburdick/myapp/cmd/myapp/app/myapp.go:25

?

@cliffburdick

This comment has been minimized.

Copy link
Author

commented Feb 6, 2019

What is your code doing here:

/home/cburdick/myapp/cmd/myapp/app/myapp.go:25

?

That's the beginning of the goroutine of the code that handles all the incoming messages using cgo:

Line 25 is the first line here:

go func(regChan chan *rpc.DpdkReg, core int) error { 
    d := dpdk.New(core, regChan, ipn)
    if err := d.DpdkInit(); err != nil {
        logrus.Fatal("Failed to initialize DPDK!")
        return nil
    }

    d.DpdkMain()

    return nil
}(regChan, dpdk_core_mask)
@randall77

This comment has been minimized.

Copy link
Contributor

commented Feb 7, 2019

Hmm, so why then is goroutine 1 stopped here:

goroutine 1 [running]:
github.com/cburdick/myapp/cmd/myapp/app.Run(0xc000185f78, 0x1)
/home/cburdick/myapp/cmd/myapp/app/myapp.go:25 +0x9a fp=0xc000185f40 sp=0xc000185ed0 pc=0xa32fda

It's just making a new goroutine.

Somewhere inside newproc we're spinning, maybe? casgstatus?
I don't understand why GOTRACEBACK=crash didn't give us more information about goroutine 1. We should have gotten the trace through the runtime for this goroutine, right?

@aclements

This comment has been minimized.

Copy link
Member

commented Feb 7, 2019

This still seems like a preemption issue, given that goroutine 5 is stuck in the call to stopTheWorldWithSema at mgc.go:1316 and that goroutine 1 is in "running" state.

Can you use delve to get a traceback of goroutine 1 (the one marked "running" in the traceback that started the goroutine)? It doesn't make sense that it would be stuck on the "go" statement in state "running". Also the output of go tool objdump -s github.com/cburdick/myapp/cmd/myapp/app.Run <binary> would be useful so we can see exactly what's at +0x9a.

Also, there are a huge number of GC worker goroutines. What's GOMAXPROCs/how many CPUs are you running with? I'm not sure if this would be relevant.

I don't understand why GOTRACEBACK=crash didn't give us more information about goroutine 1. We should have gotten the trace through the runtime for this goroutine, right?

Yes, that surprised me, too. It's also weird that it didn't traceback goroutine 5.

@cliffburdick

This comment has been minimized.

Copy link
Author

commented Feb 7, 2019

Hi @aclements, hopefully I've got all the information you need here. Here is the dlv backtrace, but I'm not sure if it's the one you wanted. The only other goroutine when I stop it in dlv is doing:

Goroutine 1 - User: .usr/local/go/src/runtime/asm_amd64.s:311 runtime.systemstack_switch (0x4e11b0)

The active one is doing this:

Goroutine 19 - User: .usr/local/go/src/fmt/print.go:128 fmt.glob..func1 (0x59f56b)

  0  0x00000000004e11b0 in runtime.systemstack_switch
    at .usr/local/go/src/runtime/asm_amd64.s:311
 1  0x00000000004a41df in runtime.gcStart
    at .usr/local/go/src/runtime/mgc.go:1316
 2  0x00000000004959e9 in runtime.mallocgc
    at .usr/local/go/src/runtime/malloc.go:996
 3  0x0000000000496158 in runtime.newobject
    at .usr/local/go/src/runtime/malloc.go:1032
 4  0x000000000059f56b in fmt.glob..func1
    at .usr/local/go/src/fmt/print.go:128
 5  0x00000000005027a8 in sync.(*Pool).Get
   at .usr/local/go/src/sync/pool.go:151
 6  0x000000000058a9ff in fmt.newPrinter
    at .usr/local/go/src/fmt/print.go:133
 7  0x000000000058b2c4 in fmt.Fprintf
    at .usr/local/go/src/fmt/print.go:187
 8  0x00000000005ef434 in github.com/sirupsen/logrus.(*TextFormatter).printColored
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/text_formatter.go:208
 9  0x00000000005ee029 in github.com/sirupsen/logrus.(*TextFormatter).Format
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/text_formatter.go:159
10  0x00000000005e7958 in github.com/sirupsen/logrus.(*Entry).write
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:154
11  0x00000000005e747a in github.com/sirupsen/logrus.Entry.log
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:130
12  0x00000000005e7ef6 in github.com/sirupsen/logrus.(*Entry).Info
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:177
13  0x00000000005e8854 in github.com/sirupsen/logrus.(*Entry).Infof
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/entry.go:221
14  0x00000000005eb351 in github.com/sirupsen/logrus.(*Logger).Infof
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/logger.go:135
15  0x00000000005e998d in github.com/sirupsen/logrus.Infof
    at .home/cburdick/go/pkg/mod/github.com/sirupsen/logrus@v1.1.1/exported.go:132
16  0x0000000000a1fed5 in github.com/cburdick/myapp/internal/dpdk.(*Dpdk).DpdkMain
    at .home/cburdick/myapp/internal/dpdk/dpdk.go:178
17  0x0000000000a33252 in github.com/cburdick/myapp/cmd/myapp/app.Run.func1
    at .home/cburdick/myapp/cmd/myapp/app/myapp.go:32
18  0x00000000004e3291 in runtime.goexit
    at .usr/local/go/src/runtime/asm_amd64.s:1333

GOMAXPROCs is the default, so I think that's the number of cores on the system, which is 80.

Here is the objdump:

└──> $ go tool objdump  -s github.com/cburdick/myapp/cmd/myapp/app.Run ~/go/bin/myapp
TEXT github.com/cburdick/myapp/cmd/myapp/app.Run(SB) /home/cburdick/myapp/cmd/myapp/app/myapp.go
  myapp.go:16          0xa32f40                64488b0c2500f0ffff      MOVQ FS:0xfffff000, CX
  myapp.go:16          0xa32f49                483b6110                CMPQ 0x10(CX), SP
  myapp.go:16          0xa32f4d                0f86f6000000            JBE 0xa33049
  myapp.go:16          0xa32f53                4883ec68                SUBQ $0x68, SP
  myapp.go:16          0xa32f57                48896c2460              MOVQ BP, 0x60(SP)
  myapp.go:16          0xa32f5c                488d6c2460              LEAQ 0x60(SP), BP
  myapp.go:16          0xa32f61                488b442468              MOVQ 0x68(SP), AX
  myapp.go:16          0xa32f66                48890424                MOVQ AX, 0(SP)
  myapp.go:16          0xa32f6a                e8411cabff              CALL runtime.racefuncenter(SB)
  myapp.go:17          0xa32f6f                e82cd8c3ff              CALL github.com/cburdick/myapp/internal/parser.ValidateCIDRFile(SB)
  myapp.go:17          0xa32f74                488b0424                MOVQ 0(SP), AX
  myapp.go:17          0xa32f78                488b4c2410              MOVQ 0x10(SP), CX
  myapp.go:17          0xa32f7d                488b542408              MOVQ 0x8(SP), DX
  myapp.go:18          0xa32f82                4885d2                  TESTQ DX, DX
  myapp.go:18          0xa32f85                7555                    JNE 0xa32fdc
  myapp.go:17          0xa32f87                4889442440              MOVQ AX, 0x40(SP)
  myapp.go:23          0xa32f8c                488d056df85a00          LEAQ 0x5af86d(IP), AX
  myapp.go:23          0xa32f93                48890424                MOVQ AX, 0(SP)
  myapp.go:23          0xa32f97                48c744240800000000      MOVQ $0x0, 0x8(SP)
  myapp.go:23          0xa32fa0                e84bb9a5ff              CALL runtime.makechan(SB)
  myapp.go:23          0xa32fa5                488b442410              MOVQ 0x10(SP), AX
  myapp.go:35          0xa32faa                488b4c2440              MOVQ 0x40(SP), CX
  myapp.go:35          0xa32faf                48894c2410              MOVQ CX, 0x10(SP)
  myapp.go:35          0xa32fb4                4889442418              MOVQ AX, 0x18(SP)
  myapp.go:35          0xa32fb9                48c744242006000000      MOVQ $0x6, 0x20(SP)
  myapp.go:25          0xa32fc2                c7042428000000          MOVL $per_lcore__thread_id.5369(SB), 0(SP)
  myapp.go:25          0xa32fc9                488d0560a06800          LEAQ 0x68a060(IP), AX
  myapp.go:25          0xa32fd0                4889442408              MOVQ AX, 0x8(SP)
  myapp.go:25          0xa32fd5                e886baa8ff              CALL runtime.newproc(SB)
  myapp.go:25          0xa32fda                ebfe                    JMP 0xa32fda
  myapp.go:17          0xa32fdc                48894c2448              MOVQ CX, 0x48(SP)
  myapp.go:17          0xa32fe1                4889542438              MOVQ DX, 0x38(SP)
  myapp.go:19          0xa32fe6                0f57c0                  XORPS X0, X0
  myapp.go:19          0xa32fe9                0f11442450              MOVUPS X0, 0x50(SP)
  myapp.go:19          0xa32fee                488d05cb505b00          LEAQ 0x5b50cb(IP), AX
  myapp.go:19          0xa32ff5                4889442450              MOVQ AX, 0x50(SP)
  myapp.go:19          0xa32ffa                488d053f0f7000          LEAQ github.com/cburdick/myapp/cmd/myapp/app.statictmp_0(SB), AX
  myapp.go:19          0xa33001                4889442458              MOVQ AX, 0x58(SP)
  myapp.go:19          0xa33006                488d442450              LEAQ 0x50(SP), AX
  myapp.go:19          0xa3300b                48890424                MOVQ AX, 0(SP)
  myapp.go:19          0xa3300f                48c744240801000000      MOVQ $0x1, 0x8(SP)
  myapp.go:19          0xa33018                48c744241001000000      MOVQ $0x1, 0x10(SP)
  myapp.go:19          0xa33021                e8ca67bbff              CALL github.com/sirupsen/logrus.Fatal(SB)
  myapp.go:20          0xa33026                e8c51babff              CALL runtime.racefuncexit(SB)
  myapp.go:20          0xa3302b                488b442438              MOVQ 0x38(SP), AX
  myapp.go:20          0xa33030                4889442470              MOVQ AX, 0x70(SP)
  myapp.go:20          0xa33035                488b442448              MOVQ 0x48(SP), AX
  myapp.go:20          0xa3303a                4889442478              MOVQ AX, 0x78(SP)
  myapp.go:20          0xa3303f                488b6c2460              MOVQ 0x60(SP), BP
  myapp.go:20          0xa33044                4883c468                ADDQ $0x68, SP
  myapp.go:20          0xa33048                c3                      RET
  myapp.go:16          0xa33049                e8c2e2aaff              CALL runtime.morestack_noctxt(SB)
  myapp.go:16          0xa3304e                e9edfeffff              JMP github.com/cburdick/myapp/cmd/myapp/app.Run(SB)

TEXT github.com/cburdick/myapp/cmd/myapp/app.Run.func1(SB) /home/cburdick/myapp/cmd/myapp/app/myapp.go
  myapp.go:25          0xa33060                64488b0c2500f0ffff      MOVQ FS:0xfffff000, CX
  myapp.go:25          0xa33069                488d442498              LEAQ -0x68(SP), AX
  myapp.go:25          0xa3306e                483b4110                CMPQ 0x10(CX), AX
  myapp.go:25          0xa33072                0f86fa010000            JBE 0xa33272
  myapp.go:25          0xa33078                4881ece8000000          SUBQ $0xe8, SP
  myapp.go:25          0xa3307f                4889ac24e0000000        MOVQ BP, 0xe0(SP)
  myapp.go:25          0xa33087                488dac24e0000000        LEAQ 0xe0(SP), BP
  myapp.go:25          0xa3308f                488b8424e8000000        MOVQ 0xe8(SP), AX
  myapp.go:25          0xa33097                48890424                MOVQ AX, 0(SP)
  myapp.go:25          0xa3309b                e8101babff              CALL runtime.racefuncenter(SB)
  dpdk.go:88            0xa330a0                e85b3ca6ff              CALL runtime.makemap_small(SB)
  dpdk.go:88            0xa330a5                488b0424                MOVQ 0(SP), AX
  dpdk.go:88            0xa330a9                4889442448              MOVQ AX, 0x48(SP)
  dpdk.go:89            0xa330ae                e84d3ca6ff              CALL runtime.makemap_small(SB)
  dpdk.go:89            0xa330b3                488b0424                MOVQ 0(SP), AX
  dpdk.go:89            0xa330b7                4889442440              MOVQ AX, 0x40(SP)
  dpdk.go:90            0xa330bc                e83f3ca6ff              CALL runtime.makemap_small(SB)
  dpdk.go:90            0xa330c1                488b0424                MOVQ 0(SP), AX
  dpdk.go:90            0xa330c5                4889442438              MOVQ AX, 0x38(SP)
  dpdk.go:91            0xa330ca                e8313ca6ff              CALL runtime.makemap_small(SB)
  dpdk.go:91            0xa330cf                488b0424                MOVQ 0(SP), AX
  dpdk.go:91            0xa330d3                4889442430              MOVQ AX, 0x30(SP)
  dpdk.go:92            0xa330d8                e8233ca6ff              CALL runtime.makemap_small(SB)
  dpdk.go:92            0xa330dd                488b0424                MOVQ 0(SP), AX
  dpdk.go:92            0xa330e1                4889442428              MOVQ AX, 0x28(SP)
  dpdk.go:93            0xa330e6                e8153ca6ff              CALL runtime.makemap_small(SB)
  dpdk.go:93            0xa330eb                488b0424                MOVQ 0(SP), AX
  dpdk.go:93            0xa330ef                4889442420              MOVQ AX, 0x20(SP)
  dpdk.go:94            0xa330f4                e8073ca6ff              CALL runtime.makemap_small(SB)
  dpdk.go:94            0xa330f9                488b0424                MOVQ 0(SP), AX
  dpdk.go:94            0xa330fd                4889442418              MOVQ AX, 0x18(SP)
  dpdk.go:95            0xa33102                e8f93ba6ff              CALL runtime.makemap_small(SB)
  dpdk.go:95            0xa33107                488b0424                MOVQ 0(SP), AX
  dpdk.go:84            0xa3310b                488d7c2460              LEAQ 0x60(SP), DI
  dpdk.go:84            0xa33110                0f57c0                  XORPS X0, X0
  dpdk.go:84            0xa33113                48896c24f0              MOVQ BP, -0x10(SP)
  dpdk.go:84            0xa33118                488d6c24f0              LEAQ -0x10(SP), BP
  dpdk.go:84            0xa3311d                e8f80babff              CALL 0x4e3d1a
  dpdk.go:84            0xa33122                488b6d00                MOVQ 0(BP), BP
  dpdk.go:84            0xa33126                488d7c2460              LEAQ 0x60(SP), DI
  dpdk.go:84            0xa3312b                48896c24f0              MOVQ BP, -0x10(SP)
  dpdk.go:84            0xa33130                488d6c24f0              LEAQ -0x10(SP), BP
  dpdk.go:84            0xa33135                e8e00babff              CALL 0x4e3d1a
  dpdk.go:84            0xa3313a                488b6d00                MOVQ 0(BP), BP
  dpdk.go:84            0xa3313e                488b8c2400010000        MOVQ 0x100(SP), CX
  dpdk.go:84            0xa33146                48894c2468              MOVQ CX, 0x68(SP)
  dpdk.go:84            0xa3314b                488b8c24f8000000        MOVQ 0xf8(SP), CX
  dpdk.go:84            0xa33153                48898c2488000000        MOVQ CX, 0x88(SP)
  dpdk.go:84            0xa3315b                488b8c24f0000000        MOVQ 0xf0(SP), CX
  dpdk.go:84            0xa33163                48898c2480000000        MOVQ CX, 0x80(SP)
  dpdk.go:84            0xa3316b                488b4c2448              MOVQ 0x48(SP), CX
  dpdk.go:84            0xa33170                48898c24a0000000        MOVQ CX, 0xa0(SP)
  dpdk.go:84            0xa33178                488b4c2440              MOVQ 0x40(SP), CX
  dpdk.go:84            0xa3317d                48898c24a8000000        MOVQ CX, 0xa8(SP)
  dpdk.go:84            0xa33185                488b4c2438              MOVQ 0x38(SP), CX
  dpdk.go:84            0xa3318a                48898c24b0000000        MOVQ CX, 0xb0(SP)
  dpdk.go:84            0xa33192                488b4c2430              MOVQ 0x30(SP), CX
  dpdk.go:84            0xa33197                48898c24c8000000        MOVQ CX, 0xc8(SP)
  dpdk.go:84            0xa3319f                488b4c2428              MOVQ 0x28(SP), CX
  dpdk.go:84            0xa331a4                48898c24b8000000        MOVQ CX, 0xb8(SP)
  dpdk.go:84            0xa331ac                488b4c2420              MOVQ 0x20(SP), CX
  dpdk.go:84            0xa331b1                48898c24c0000000        MOVQ CX, 0xc0(SP)
  dpdk.go:84            0xa331b9                488b4c2418              MOVQ 0x18(SP), CX
  dpdk.go:84            0xa331be                48898c24d8000000        MOVQ CX, 0xd8(SP)
  dpdk.go:84            0xa331c6                48898424d0000000        MOVQ AX, 0xd0(SP)
  myapp.go:27          0xa331ce                488d442460              LEAQ 0x60(SP), AX
  myapp.go:27          0xa331d3                48890424                MOVQ AX, 0(SP)
  myapp.go:27          0xa331d7                e894befeff              CALL github.com/cburdick/myapp/internal/dpdk.(*Dpdk).DpdkInit(SB)
  myapp.go:27          0xa331dc                48837c240800            CMPQ $0x0, 0x8(SP)
  myapp.go:27          0xa331e2                7460                    JE 0xa33244
  myapp.go:28          0xa331e4                0f57c0                  XORPS X0, X0
  myapp.go:28          0xa331e7                0f11442450              MOVUPS X0, 0x50(SP)
  myapp.go:28          0xa331ec                488d05cd4e5b00          LEAQ 0x5b4ecd(IP), AX
  myapp.go:28          0xa331f3                4889442450              MOVQ AX, 0x50(SP)
  myapp.go:28          0xa331f8                488d05510d7000          LEAQ github.com/cburdick/myapp/cmd/myapp/app.statictmp_1(SB), AX
  myapp.go:28          0xa331ff                4889442458              MOVQ AX, 0x58(SP)
  myapp.go:28          0xa33204                488d442450              LEAQ 0x50(SP), AX
  myapp.go:28          0xa33209                48890424                MOVQ AX, 0(SP)
  myapp.go:28          0xa3320d                48c744240801000000      MOVQ $0x1, 0x8(SP)
  myapp.go:28          0xa33216                48c744241001000000      MOVQ $0x1, 0x10(SP)
  myapp.go:28          0xa3321f                e8cc65bbff              CALL github.com/sirupsen/logrus.Fatal(SB)
  myapp.go:29          0xa33224                e8c719abff              CALL runtime.racefuncexit(SB)
  myapp.go:29          0xa33229                0f57c0                  XORPS X0, X0
  myapp.go:29          0xa3322c                0f11842408010000        MOVUPS X0, 0x108(SP)
  myapp.go:29          0xa33234                488bac24e0000000        MOVQ 0xe0(SP), BP
  myapp.go:29          0xa3323c                4881c4e8000000          ADDQ $0xe8, SP
  myapp.go:29          0xa33243                c3                      RET
  myapp.go:32          0xa33244                488d442460              LEAQ 0x60(SP), AX
  myapp.go:32          0xa33249                48890424                MOVQ AX, 0(SP)
  myapp.go:32          0xa3324d                e89ec9feff              CALL github.com/cburdick/myapp/internal/dpdk.(*Dpdk).DpdkMain(SB)
  myapp.go:34          0xa33252                e89919abff              CALL runtime.racefuncexit(SB)
  myapp.go:34          0xa33257                0f57c0                  XORPS X0, X0
  myapp.go:34          0xa3325a                0f11842408010000        MOVUPS X0, 0x108(SP)
  myapp.go:34          0xa33262                488bac24e0000000        MOVQ 0xe0(SP), BP
  myapp.go:34          0xa3326a                4881c4e8000000          ADDQ $0xe8, SP
  myapp.go:34          0xa33271                c3                      RET
  myapp.go:25          0xa33272                e899e0aaff              CALL runtime.morestack_noctxt(SB)
  myapp.go:25          0xa33277                e9e4fdffff              JMP github.com/cburdick/myapp/cmd/myapp/app.Run.func1(SB)
(env)
@aclements

This comment has been minimized.

Copy link
Member

commented Feb 7, 2019

Thanks. Based on the objdump, we're in the obvious place: at the call to newproc at PC 0xa32fd5. The traceback has been truncated since +0x9a is the return PC from that call.

@cliffburdick, thanks for the traceback, but I actually wanted to see the Delve traceback of goroutine 1. Though given the inner-most frame you mentioned, which is in systemstack_switch, I suspect that's not going to tell us what we need to know, but it might.

Can you attach gdb to it? If so, it would be great to see the output of "thread apply all bt". That might be huge with GOMAXPROCS=80, so if you can reproduce with a smaller GOMAXPROCS I would try that.

@cliffburdick

This comment has been minimized.

Copy link
Author

commented Feb 18, 2019

@aclements I'll get back to this shortly, hopefully. Got pulled onto something else temporarily.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 7, 2019

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this Apr 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.