Skip to content

/ go

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: panic: non-empty mark queue after concurrent mark #41303

Open
navytux opened this issue Sep 9, 2020 · 2 comments
Open

runtime: panic: non-empty mark queue after concurrent mark #41303

navytux opened this issue Sep 9, 2020 · 2 comments
Labels
NeedsInvestigation
Milestone
Go1.16

Comments

@navytux
Copy link
Contributor

@navytux navytux commented Sep 9, 2020 •
edited

Hello up there.

Today, while working on wendelin.core I've got panic: non-empty mark queue after concurrent mark several times while running unit tests. It is hard for me to find time to try to reduce the problem to minimal example, but I just cannot ignore such garbage collector kind of bug, so I decided to report at least what I can. Please find details about GC crash below. I also attach dumped core in case it could be useful to analyse what was going on (no confidential data in there).

Thanks beforehand,
Kirill

What version of Go are you using (go version)?

$ go version
go version go1.15.2 linux/amd64

Does this issue reproduce with the latest release?

No data

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/kirr/.cache/go-build"
GOENV="/home/kirr/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/kirr/src/neo/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/kirr/src/neo:/home/kirr/src/tools/go/g.env"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/kirr/src/tools/go/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/kirr/src/tools/go/go/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build670581309=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.15.2 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.15.2
uname -sr: Linux 5.7.0-3-amd64
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	testing
Codename:	bullseye
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Debian GLIBC 2.31-3) stable release version 2.31.
gdb --version: GNU gdb (Debian 9.2-1) 9.2

What did you do?

What did you expect to see?

No Go runtime crashes

What did you see instead?

Go runtime crashed with

runtime: full=0xc0003d3800000e next=167 jobs=164 nDataRoots=1 nBSSRoots=1 nSpanRoots=16 nStackRoots=144
panic: non-empty mark queue after concurrent mark
fatal error: panic on system stack
full output 

...
=== RUN   TestZBlk
2020/09/09 22:29:15 zodb: FIXME: open testdata/zblk.fs: raw cache is not ready for invalidations -> NoCache forced
--- PASS: TestZBlk (0.02s)
=== RUN   TestΔBTail
2020/09/09 22:29:16 zodb: FIXME: open /tmp/δBTail797036932/1.fs: raw cache is not ready for invalidations -> NoCache forced
runtime: full=0xc0003d3800000e next=167 jobs=164 nDataRoots=1 nBSSRoots=1 nSpanRoots=16 nStackRoots=144
panic: non-empty mark queue after concurrent mark
fatal error: panic on system stack

runtime stack:
runtime.throw(0x79c93c, 0x15)
        /home/kirr/src/tools/go/go/src/runtime/panic.go:1116 +0x72 fp=0x7f0ea311cd10 sp=0x7f0ea311cce0 pc=0x43abf2
panic(0x70d360, 0x7f4070)
        /home/kirr/src/tools/go/go/src/runtime/panic.go:895 +0x750 fp=0x7f0ea311cdc8 sp=0x7f0ea311cd10 pc=0x43aaf0
runtime.gcMark(0x2036b5d1c005)
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:2064 +0x539 fp=0x7f0ea311ce60 sp=0x7f0ea311cdc8 pc=0x421239
runtime.gcMarkTermination.func1()
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1656 +0x2a fp=0x7f0ea311ce78 sp=0x7f0ea311ce60 pc=0x46868a
runtime.systemstack(0x7f0e98000020)
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:370 +0x66 fp=0x7f0ea311ce80 sp=0x7f0ea311ce78 pc=0x4701a6
runtime.mstart()
        /home/kirr/src/tools/go/go/src/runtime/proc.go:1116 fp=0x7f0ea311ce88 sp=0x7f0ea311ce80 pc=0x43fdc0
goroutine 99 [garbage collection]:
runtime.systemstack_switch()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:330 fp=0xc00002e548 sp=0xc00002e540 pc=0x470120
runtime.gcMarkTermination(0x3ff699bb8842651e)
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1655 +0x17b fp=0xc00002e708 sp=0xc00002e548 pc=0x41fcfb
runtime.gcMarkDone()
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1630 +0x275 fp=0xc00002e760 sp=0xc00002e708 pc=0x41fab5
runtime.gcBgMarkWorker(0xc00002b800)
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:2018 +0x2af fp=0xc00002e7d8 sp=0xc00002e760 pc=0x420acf
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00002e7e0 sp=0xc00002e7d8 pc=0x471f01
created by runtime.gcBgMarkStartWorkers
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1839 +0x77

goroutine 1 [chan receive]:
runtime.gopark(0x7af8c0, 0xc000224118, 0x50170e, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc0000a4b58 sp=0xc0000a4b38 pc=0x43d7e5
runtime.chanrecv(0xc0002240c0, 0xc0000a4c57, 0xc000000101, 0x5032e6)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:567 +0x365 fp=0xc0000a4be8 sp=0xc0000a4b58 pc=0x409d45
runtime.chanrecv1(0xc0002240c0, 0xc0000a4c57)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:434 +0x2b fp=0xc0000a4c18 sp=0xc0000a4be8 pc=0x40998b
testing.(*T).Run(0xc000082d80, 0x798700, 0xb, 0x7aeb88, 0x493c01)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1179 +0x3ad fp=0xc0000a4cb0 sp=0xc0000a4c18 pc=0x50330d
testing.runTests.func1(0xc000001b00)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1449 +0x78 fp=0xc0000a4d00 sp=0xc0000a4cb0 pc=0x507718
testing.tRunner(0xc000001b00, 0xc000141de0)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1127 +0xef fp=0xc0000a4d50 sp=0xc0000a4d00 pc=0x502ecf
testing.runTests(0xc00000e9c0, 0x99f2a0, 0x6, 0x6, 0xbfce69b8da4fa4a3, 0x8bb2dad0e0, 0x9d3580, 0x410e30)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1447 +0x2e8 fp=0xc0000a4e10 sp=0xc0000a4d50 pc=0x504b28
testing.(*M).Run(0xc00015c080, 0x0)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1357 +0x245 fp=0xc0000a4f20 sp=0xc0000a4e10 pc=0x5038c5
main.main()
        _testmain.go:53 +0x138 fp=0xc0000a4f88 sp=0xc0000a4f20 pc=0x6dd4b8
runtime.main()
        /home/kirr/src/tools/go/go/src/runtime/proc.go:204 +0x209 fp=0xc0000a4fe0 sp=0xc0000a4f88 pc=0x43d3e9
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000a4fe8 sp=0xc0000a4fe0 pc=0x471f01

goroutine 2 [force gc (idle)]:
runtime.gopark(0x7afad8, 0x9d2c00, 0x1411, 0x1)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000032fb0 sp=0xc000032f90 pc=0x43d7e5
runtime.goparkunlock(...)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:312
runtime.forcegchelper()
        /home/kirr/src/tools/go/go/src/runtime/proc.go:255 +0xc5 fp=0xc000032fe0 sp=0xc000032fb0 pc=0x43d685
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000032fe8 sp=0xc000032fe0 pc=0x471f01
created by runtime.init.6
        /home/kirr/src/tools/go/go/src/runtime/proc.go:243 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x7afad8, 0x9d2dc0, 0x140c, 0x1)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc0000337a8 sp=0xc000033788 pc=0x43d7e5
runtime.goparkunlock(...)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:312
runtime.bgsweep(0xc00005a000)
        /home/kirr/src/tools/go/go/src/runtime/mgcsweep.go:182 +0x13b fp=0xc0000337d8 sp=0xc0000337a8 pc=0x427f1b
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000337e0 sp=0xc0000337d8 pc=0x471f01
created by runtime.gcenable
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:217 +0x5c

goroutine 4 [sleep]:
runtime.gopark(0x7afad8, 0x9d31e0, 0x1313, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000033f20 sp=0xc000033f00 pc=0x43d7e5
runtime.goparkunlock(...)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:312
runtime.scavengeSleep(0x11015e, 0x40804)
        /home/kirr/src/tools/go/go/src/runtime/mgcscavenge.go:241 +0xc5 fp=0xc000033f78 sp=0xc000033f20 pc=0x425da5
runtime.bgscavenge(0xc00005a000)
        /home/kirr/src/tools/go/go/src/runtime/mgcscavenge.go:366 +0x1e5 fp=0xc000033fd8 sp=0xc000033f78 pc=0x425fc5
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000033fe0 sp=0xc000033fd8 pc=0x471f01
created by runtime.gcenable
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:218 +0x7e

goroutine 5 [finalizer wait]:
runtime.gopark(0x7afad8, 0xa05678, 0xc000221410, 0x1)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000034758 sp=0xc000034738 pc=0x43d7e5
runtime.goparkunlock(...)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:312
runtime.runfinq()
        /home/kirr/src/tools/go/go/src/runtime/mfinal.go:175 +0xa9 fp=0xc0000347e0 sp=0xc000034758 pc=0x41cca9
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000347e8 sp=0xc0000347e0 pc=0x471f01
created by runtime.createfing
        /home/kirr/src/tools/go/go/src/runtime/mfinal.go:156 +0x65

goroutine 6 [chan receive]:
runtime.gopark(0x7af8c0, 0xc00008e058, 0x170e, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc0000326d0 sp=0xc0000326b0 pc=0x43d7e5
runtime.chanrecv(0xc00008e000, 0xc0000327b0, 0xc000092001, 0xc00008e000)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:567 +0x365 fp=0xc000032760 sp=0xc0000326d0 pc=0x409d45
runtime.chanrecv2(0xc00008e000, 0xc0000327b0, 0x0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:439 +0x2b fp=0xc000032790 sp=0xc000032760 pc=0x4099cb
github.com/golang/glog.(*loggingT).flushDaemon(0x9d3840)
        /home/kirr/src/neo/src/github.com/golang/glog/glog.go:882 +0x8b fp=0xc0000327d8 sp=0xc000032790 pc=0x548d4b
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000327e0 sp=0xc0000327d8 pc=0x471f01
created by github.com/golang/glog.init.0
        /home/kirr/src/neo/src/github.com/golang/glog/glog.go:410 +0x274

goroutine 83 [semacquire]:
runtime.gopark(0x7afad8, 0x9da260, 0xc000441912, 0x4)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc0002def08 sp=0xc0002deee8 pc=0x43d7e5
runtime.goparkunlock(...)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:312
runtime.semacquire1(0xc0004fc668, 0x445800, 0x1, 0x0)
        /home/kirr/src/tools/go/go/src/runtime/sema.go:144 +0x1c5 fp=0xc0002def70 sp=0xc0002def08 pc=0x44e185
sync.runtime_Semacquire(0xc0004fc668)
        /home/kirr/src/tools/go/go/src/runtime/sema.go:56 +0x45 fp=0xc0002defa0 sp=0xc0002def70 pc=0x46e5c5
sync.(*WaitGroup).Wait(0xc0004fc660)
        /home/kirr/src/tools/go/go/src/sync/waitgroup.go:130 +0x65 fp=0xc0002defc8 sp=0xc0002defa0 pc=0x47c7e5
lab.nexedi.com/kirr/neo/go/transaction.(*transaction).Abort(0xc000120540)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/transaction/transaction.go:120 +0x113 fp=0xc0002df040 sp=0xc0002defc8 pc=0x56a2b3
_/home/kirr/src/wendelin/wendelin.core/wcfs.(*ΔBtail).Update(0xc0002df4a8, 0xc000345b80, 0x3da809146177299, 0xc0003aa9c0, 0x0, 0x0)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail.go:577 +0x5d5 fp=0xc0002df1d8 sp=0xc0002df040 pc=0x6b4235
_/home/kirr/src/wendelin/wendelin.core/wcfs.xverifyΔBTail1(0xc000082d80, 0xc0001f8200, 0x38, 0xc00011c200, 0xc, 0x3da80914603abcc, 0x3da809146177299, 0xc0002d00f8, 0x1, 0x1, ...)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:695 +0x9d1 fp=0xc0002df668 sp=0xc0002df1d8 pc=0x6c3bb1
_/home/kirr/src/wendelin/wendelin.core/wcfs.xverifyΔBTail(0xc000082d80, 0xc0001f8200, 0x38, 0xc00011c200, 0xc, 0x3da80914603abcc, 0x3da809146177299, 0xc0002d00f8, 0x1, 0x1, ...)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:599 +0xeff fp=0xc0002dfb28 sp=0xc0002df668 pc=0x6c2fff
_/home/kirr/src/wendelin/wendelin.core/wcfs.testΔBTail(0xc000082d80, 0xc000224240)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:854 +0x5ff fp=0xc0002dfd38 sp=0xc0002dfb28 pc=0x6c5b7f
_/home/kirr/src/wendelin/wendelin.core/wcfs.TestΔBTail(0xc000082d80)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1081 +0x3408 fp=0xc0002dff80 sp=0xc0002dfd38 pc=0x6c9108
testing.tRunner(0xc000082d80, 0x7aeb88)
        /home/kirr/src/tools/go/go/src/testing/testing.go:1127 +0xef fp=0xc0002dffd0 sp=0xc0002dff80 pc=0x502ecf
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0002dffd8 sp=0xc0002dffd0 pc=0x471f01
created by testing.(*T).Run
        /home/kirr/src/tools/go/go/src/testing/testing.go:1178 +0x386

goroutine 84 [chan send]:
runtime.gopark(0x7af8c0, 0xc000224298, 0xc00003160f, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000035690 sp=0xc000035670 pc=0x43d7e5
runtime.chansend(0xc000224240, 0xc000035788, 0x46f001, 0x6cf1e7, 0x1bbd5c01)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:253 +0x23c fp=0xc000035710 sp=0xc000035690 pc=0x40907c
runtime.chansend1(0xc000224240, 0xc000035788)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:143 +0x35 fp=0xc000035748 sp=0xc000035710 pc=0x408e35
_/home/kirr/src/wendelin/wendelin.core/wcfs.TestΔBTail.func3(0xc000224240, 0xc000160700, 0x69, 0x70)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1078 +0xc7 fp=0xc0000357c0 sp=0xc000035748 pc=0x6cf1e7
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000357c8 sp=0xc0000357c0 pc=0x471f01
created by _/home/kirr/src/wendelin/wendelin.core/wcfs.TestΔBTail
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1075 +0x33ed

goroutine 17451 [runnable]:
lab.nexedi.com/kirr/neo/go/transaction.(*transaction).Abort.func2(0xc0004fc660, 0xc0004ff560, 0x0, 0xc000120540)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/transaction/transaction.go:113 fp=0xc00002ffc0 sp=0xc00002ffb8 pc=0x56aac0
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00002ffc8 sp=0xc00002ffc0 pc=0x471f01
created by lab.nexedi.com/kirr/neo/go/transaction.(*transaction).Abort
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/transaction/transaction.go:113 +0xf1

goroutine 82 [GC worker (idle)]:
runtime.gopark(0x7af960, 0xc000017ba0, 0x1418, 0x0)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000035f60 sp=0xc000035f40 pc=0x43d7e5
runtime.gcBgMarkWorker(0xc000024000)
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1891 +0xff fp=0xc000035fd8 sp=0xc000035f60 pc=0x42091f
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000035fe0 sp=0xc000035fd8 pc=0x471f01
created by runtime.gcBgMarkStartWorkers
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1839 +0x77

goroutine 17240 [chan send]:
runtime.gopark(0x7af8c0, 0xc000439fd8, 0x7f0ecad4160f, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000276e50 sp=0xc000276e30 pc=0x43d7e5
runtime.chansend(0xc000439f80, 0xc000276fa0, 0x70ed01, 0x6ca5bd, 0xc0004167c0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:253 +0x23c fp=0xc000276ed0 sp=0xc000276e50 pc=0x40907c
runtime.chansend1(0xc000439f80, 0xc000276fa0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:143 +0x35 fp=0xc000276f08 sp=0xc000276ed0 pc=0x408e35
_/home/kirr/src/wendelin/wendelin.core/wcfs.intSets(0xc000439f80, 0x2, 0x8)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1197 +0xdd fp=0xc000276fc8 sp=0xc000276f08 pc=0x6ca5bd
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000276fd0 sp=0xc000276fc8 pc=0x471f01
created by _/home/kirr/src/wendelin/wendelin.core/wcfs.intSets
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1195 +0x229

goroutine 114 [runnable]:
runtime.gopark(0x7afad8, 0x9d8be0, 0xc000441912, 0x4)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc00025bea0 sp=0xc00025be80 pc=0x43d7e5
runtime.goparkunlock(...)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:312
runtime.semacquire1(0x9d3e94, 0xbfa5398e8180a000, 0x0, 0x0)
        /home/kirr/src/tools/go/go/src/runtime/sema.go:144 +0x1c5 fp=0xc00025bf08 sp=0xc00025bea0 pc=0x44e185
runtime.semacquire(...)
        /home/kirr/src/tools/go/go/src/runtime/sema.go:95
runtime.gcMarkDone()
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1447 +0x3e fp=0xc00025bf60 sp=0xc00025bf08 pc=0x41f87e
runtime.gcBgMarkWorker(0xc000029000)
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:2018 +0x2af fp=0xc00025bfd8 sp=0xc00025bf60 pc=0x420acf
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00025bfe0 sp=0xc00025bfd8 pc=0x471f01
created by runtime.gcBgMarkStartWorkers
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1839 +0x77

goroutine 98 [GC worker (idle)]:
runtime.gopark(0x7af960, 0xc0000b6160, 0x1418, 0x0)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000255760 sp=0xc000255740 pc=0x43d7e5
runtime.gcBgMarkWorker(0xc000026800)
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1891 +0xff fp=0xc0002557d8 sp=0xc000255760 pc=0x42091f
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0002557e0 sp=0xc0002557d8 pc=0x471f01
created by runtime.gcBgMarkStartWorkers
        /home/kirr/src/tools/go/go/src/runtime/mgc.go:1839 +0x77

goroutine 17452 [runnable]:
lab.nexedi.com/kirr/neo/go/transaction.(*transaction).Abort.func2(0xc0004fc660, 0xc0004ff560, 0x1, 0xc000120540)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/transaction/transaction.go:113 fp=0xc0005b6fc0 sp=0xc0005b6fb8 pc=0x56aac0
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0005b6fc8 sp=0xc0005b6fc0 pc=0x471f01
created by lab.nexedi.com/kirr/neo/go/transaction.(*transaction).Abort
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/transaction/transaction.go:113 +0xf1

goroutine 15832 [chan send]:
runtime.gopark(0x7af8c0, 0xc0002251f8, 0x7f0ecad4160f, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc0005b7650 sp=0xc0005b7630 pc=0x43d7e5
runtime.chansend(0xc0002251a0, 0xc0005b77a0, 0x70ed01, 0x6ca5bd, 0xc0004545d0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:253 +0x23c fp=0xc0005b76d0 sp=0xc0005b7650 pc=0x40907c
runtime.chansend1(0xc0002251a0, 0xc0005b77a0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:143 +0x35 fp=0xc0005b7708 sp=0xc0005b76d0 pc=0x408e35
_/home/kirr/src/wendelin/wendelin.core/wcfs.intSets(0xc0002251a0, 0x0, 0x8)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1197 +0xdd fp=0xc0005b77c8 sp=0xc0005b7708 pc=0x6ca5bd
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0005b77d0 sp=0xc0005b77c8 pc=0x471f01
created by _/home/kirr/src/wendelin/wendelin.core/wcfs.IntSets
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1184 +0x6b

goroutine 153 [select]:
runtime.gopark(0x7afb28, 0x0, 0x1809, 0x1)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000215858 sp=0xc000215838 pc=0x43d7e5
runtime.selectgo(0xc000215e80, 0xc000215a50, 0x4, 0xc000048001, 0x7fa100)
        /home/kirr/src/tools/go/go/src/runtime/select.go:319 +0xce5 fp=0xc000215980 sp=0xc000215858 pc=0x44d7c5
lab.nexedi.com/kirr/neo/go/zodb/storage/fs1.(*FileStorage)._watcher(0xc00011e000, 0xc0000920f0, 0x0, 0x0, 0x0)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/filestorage.go:537 +0x1073 fp=0xc000215f30 sp=0xc000215980 pc=0x5b4cd3
lab.nexedi.com/kirr/neo/go/zodb/storage/fs1.(*FileStorage).watcher(0xc00011e000, 0xc0000920f0, 0x0)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/filestorage.go:465 +0xa5 fp=0xc000215fc8 sp=0xc000215f30 pc=0x5b3985
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000215fd0 sp=0xc000215fc8 pc=0x471f01
created by lab.nexedi.com/kirr/neo/go/zodb/storage/fs1.Open
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/fs1/filestorage.go:906 +0x6a5

goroutine 155 [select]:
runtime.gopark(0x7afb28, 0x0, 0x1809, 0x1)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000046cf8 sp=0xc000046cd8 pc=0x43d7e5
runtime.selectgo(0xc000046ef0, 0xc000046e88, 0x2, 0x3da808746177299, 0x197e7162)
        /home/kirr/src/tools/go/go/src/runtime/select.go:319 +0xce5 fp=0xc000046e20 sp=0xc000046cf8 pc=0x44d7c5
lab.nexedi.com/kirr/neo/go/zodb.(*DB).watcher(0xc00011c200, 0x0, 0x0)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/db.go:234 +0x3b4 fp=0xc000046fc8 sp=0xc000046e20 pc=0x594db4
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000046fd0 sp=0xc000046fc8 pc=0x471f01
created by lab.nexedi.com/kirr/neo/go/zodb.NewDB
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/db.go:142 +0x167

goroutine 154 [select]:
runtime.gopark(0x7afb28, 0x0, 0x1809, 0x1)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc000043c10 sp=0xc000043bf0 pc=0x43d7e5
runtime.selectgo(0xc000043ed8, 0xc000043d98, 0x2, 0x0, 0x1)
        /home/kirr/src/tools/go/go/src/runtime/select.go:319 +0xce5 fp=0xc000043d38 sp=0xc000043c10 pc=0x44d7c5
lab.nexedi.com/kirr/neo/go/zodb.(*storage)._watcher(0xc00011c180, 0x0, 0x0)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage.go:310 +0x1ce fp=0xc000043fb0 sp=0xc000043d38 pc=0x59d02e
lab.nexedi.com/kirr/neo/go/zodb.(*storage).watcher(0xc00011c180)
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage.go:249 +0x2b fp=0xc000043fd8 sp=0xc000043fb0 pc=0x59ce2b
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000043fe0 sp=0xc000043fd8 pc=0x471f01
created by lab.nexedi.com/kirr/neo/go/zodb.Open
        /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage.go:148 +0x23e

goroutine 152 [syscall]:
syscall.Syscall6(0xe8, 0xb, 0xc00018fb6c, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x2, 0x4, 0x0)
        /home/kirr/src/tools/go/go/src/syscall/asm_linux_amd64.s:41 +0x5 fp=0xc00018faa0 sp=0xc00018fa98 pc=0x4871e5
golang.org/x/sys/unix.EpollWait(0xb, 0xc00018fb6c, 0x7, 0x7, 0xffffffffffffffff, 0xc00018fc44, 0xc000552d00, 0x0)
        /home/kirr/src/tools/go/g.env/src/golang.org/x/sys/unix/zsyscall_linux_amd64.go:76 +0x72 fp=0xc00018fb10 sp=0xc00018faa0 pc=0x54fc92
github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc0001fa100, 0xc00018fc00, 0x2, 0x0)
        /home/kirr/src/neo/src/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x91 fp=0xc00018fbd8 sp=0xc00018fb10 pc=0x5ab551
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0000920f0)
        /home/kirr/src/neo/src/github.com/fsnotify/fsnotify/inotify.go:192 +0x206 fp=0xc00019ffd8 sp=0xc00018fbd8 pc=0x5aa6c6
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00019ffe0 sp=0xc00019ffd8 pc=0x471f01
created by github.com/fsnotify/fsnotify.NewWatcher
        /home/kirr/src/neo/src/github.com/fsnotify/fsnotify/inotify.go:59 +0x1a8

goroutine 17413 [chan send]:
runtime.gopark(0x7af8c0, 0xc00022f918, 0x10000000000160f, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc0001a5e50 sp=0xc0001a5e30 pc=0x43d7e5
runtime.chansend(0xc00022f8c0, 0xc0001a5fa0, 0x70ed01, 0x6ca5bd, 0xc0000b7080)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:253 +0x23c fp=0xc0001a5ed0 sp=0xc0001a5e50 pc=0x40907c
runtime.chansend1(0xc00022f8c0, 0xc0001a5fa0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:143 +0x35 fp=0xc0001a5f08 sp=0xc0001a5ed0 pc=0x408e35
_/home/kirr/src/wendelin/wendelin.core/wcfs.intSets(0xc00022f8c0, 0x5, 0x8)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1197 +0xdd fp=0xc0001a5fc8 sp=0xc0001a5f08 pc=0x6ca5bd
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0001a5fd0 sp=0xc0001a5fc8 pc=0x471f01
created by _/home/kirr/src/wendelin/wendelin.core/wcfs.intSets
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1195 +0x229

goroutine 17241 [chan send]:
runtime.gopark(0x7af8c0, 0xc000496058, 0x7f0ecad4160f, 0x2)
        /home/kirr/src/tools/go/go/src/runtime/proc.go:306 +0xe5 fp=0xc0001a4650 sp=0xc0001a4630 pc=0x43d7e5
runtime.chansend(0xc000496000, 0xc0001a47a0, 0x70ed01, 0x6ca5bd, 0xc0004167e0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:253 +0x23c fp=0xc0001a46d0 sp=0xc0001a4650 pc=0x40907c
runtime.chansend1(0xc000496000, 0xc0001a47a0)
        /home/kirr/src/tools/go/go/src/runtime/chan.go:143 +0x35 fp=0xc0001a4708 sp=0xc0001a46d0 pc=0x408e35
_/home/kirr/src/wendelin/wendelin.core/wcfs.intSets(0xc000496000, 0x3, 0x8)
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1197 +0xdd fp=0xc0001a47c8 sp=0xc0001a4708 pc=0x6ca5bd
runtime.goexit()
        /home/kirr/src/tools/go/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0001a47d0 sp=0xc0001a47c8 pc=0x471f01
created by _/home/kirr/src/wendelin/wendelin.core/wcfs.intSets
        /home/kirr/src/wendelin/wendelin.core/wcfs/δbtail_test.go:1195 +0x229
signal: aborted (core dumped)
FAIL    _/home/kirr/src/wendelin/wendelin.core/wcfs     1.114s

core.zip
@navytux
Copy link
Contributor Author

@navytux navytux commented Sep 9, 2020 •
edited

( wcfs go unit tests are just regular go unit tests - they do not play with virtual memory at all, , and, to my knowledge, there is no Cgo involved. At that level FUSE is also not used. )
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 9, 2020

CC @aclements @mknyszek
@ianlancetaylor ianlancetaylor added this to the Go1.16 milestone Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation
Projects
None yet
Milestone
Go1.16
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@navytux @ianlancetaylor
You can’t perform that action at this time.