Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: sweep increased allocation count (With go get) #19029

Closed
cynecx opened this issue Feb 10, 2017 · 56 comments

Comments

Projects
None yet
@cynecx
Copy link

commented Feb 10, 2017

What version of Go are you using (go version)?

go version go1.7.5 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/cynecx/dev/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build806428553=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

go get -u -v github.com/nsf/gocode

What did you expect to see?

Anything except runtime errors.

What did you see instead?

runtime: nelems=256 nfree=0 nalloc=256 previous allocCount=254 nfreed=65534
fatal error: sweep increased allocation count

runtime stack:
runtime.throw(0x856aef, 0x20)
    /usr/lib/go/src/runtime/panic.go:566 +0x95
runtime.(*mspan).sweep(0x7f699add7b20, 0x500000000, 0xc400000001)
    /usr/lib/go/src/runtime/mgcsweep.go:287 +0x7ab
runtime.sweepone(0x4a8bf6)
    /usr/lib/go/src/runtime/mgcsweep.go:112 +0xfe
runtime.gosweepone.func1()
    /usr/lib/go/src/runtime/mgcsweep.go:124 +0x2b
runtime.systemstack(0xc420020000)
    /usr/lib/go/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
    /usr/lib/go/src/runtime/proc.go:1079

goroutine 3 [running]:
runtime.systemstack_switch()
    /usr/lib/go/src/runtime/asm_amd64.s:252 fp=0xc420030f48 sp=0xc420030f40
runtime.gosweepone(0x0)
    /usr/lib/go/src/runtime/mgcsweep.go:125 +0x53 fp=0xc420030f78 sp=0xc420030f48
runtime.bgsweep(0xc420060000)
    /usr/lib/go/src/runtime/mgcsweep.go:66 +0xbb fp=0xc420030fb8 sp=0xc420030f78
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420030fc0 sp=0xc420030fb8
created by runtime.gcenable
    /usr/lib/go/src/runtime/mgc.go:195 +0x61

goroutine 1 [runnable]:
os.OpenFile(0xc4200159b0, 0x24, 0x0, 0xc400000000, 0x2, 0x2, 0xc4200159b0)
    /usr/lib/go/src/os/file_unix.go:86
os.Open(0xc4200159b0, 0x24, 0x2, 0x2, 0xc4200159b0)
    /usr/lib/go/src/os/file.go:250 +0x46
go/build.(*Context).openFile(0xa2ea40, 0xc4200159b0, 0x24, 0x2, 0xc4200159b0, 0x24, 0x26eb)
    /usr/lib/go/src/go/build/build.go:184 +0x94
go/build.(*Context).matchFile(0xa2ea40, 0xc4203871c0, 0x13, 0xc42033c6a4, 0x10, 0xc420014c01, 0xc42043cbf0, 0xc4201d0b60, 0x0, 0xc420382000, ...)
    /usr/lib/go/src/go/build/build.go:1045 +0x2fd
go/build.(*Context).Import(0xa2ea40, 0xc42019f821, 0x3, 0xc42017d200, 0x2d, 0x4, 0xc4201fcd01, 0xc42043cf50, 0x47fc61)
    /usr/lib/go/src/go/build/build.go:690 +0x9ec
main.loadImport(0xc42019f821, 0x3, 0xc42017d200, 0x2d, 0xc4201b2000, 0xc4201a8180, 0xc4201c70b0, 0x1, 0x1, 0x1, ...)
    /usr/lib/go/src/cmd/go/pkg.go:365 +0x3b9
main.(*Package).load(0xc4201b2000, 0xc4201a8180, 0xc4201a6a80, 0x0, 0x0, 0xc)
    /usr/lib/go/src/cmd/go/pkg.go:940 +0x12b6
main.loadImport(0x7ffd56c3b483, 0x15, 0xc420010064, 0x13, 0x0, 0xc4201a8180, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/lib/go/src/cmd/go/pkg.go:374 +0x470
main.loadPackage(0x7ffd56c3b483, 0x15, 0xc4201a8180, 0x0)
    /usr/lib/go/src/cmd/go/pkg.go:1658 +0x28f
main.download.func1(0x7ffd56c3b483, 0x15, 0x0, 0xc42019e6e0)
    /usr/lib/go/src/cmd/go/get.go:190 +0xba
main.download(0x7ffd56c3b483, 0x15, 0x0, 0xc4201a8180, 0x0)
    /usr/lib/go/src/cmd/go/get.go:195 +0xf3
main.runGet(0xa27900, 0xc42000c0e0, 0x1, 0x1)
    /usr/lib/go/src/cmd/go/get.go:106 +0x145
main.main()
    /usr/lib/go/src/cmd/go/main.go:181 +0x624

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
    /usr/lib/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/lib/go/src/os/signal/signal_unix.go:28 +0x41

I am currently using a customized (ck/muqss+bfq) kernel but I don't think this is the issue because it is also crashing with a freshly compiled vanilla linux kernel (v4.9.9).

@bradfitz bradfitz changed the title Fatal error: sweep increased allocation count (With go get) runtime: fatal error: sweep increased allocation count (With go get) Feb 10, 2017

@bradfitz

This comment has been minimized.

Copy link
Member

commented Feb 10, 2017

@aclements, I forget whether this was ever debugged. There was also #16778.

@brauner

This comment has been minimized.

Copy link

commented Feb 12, 2017

@bradfitz, we're seeing the same issue (not with go get) in https://github.com/lxc/lxd. We're actually not sure whether this is a golang regression but the interesting observation is that we only see this on or golang tip test runners. Neither golang 1.6 nor golang 1.7 seem to trigger this error.

Stacktrace:

runtime: nelems=128 nfree=86 nalloc=42 previous allocCount=40 nfreed=65534
fatal error: sweep increased allocation count

runtime stack:
runtime.throw(0x85df560, 0x20)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/panic.go:596 +0x7c
runtime.(*mspan).sweep(0x86d72284, 0x0, 0xf)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/mgcsweep.go:288 +0x7b5
runtime.sweepone(0x807a1bc)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/mgcsweep.go:110 +0x1ab
runtime.gosweepone.func1()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/mgcsweep.go:125 +0x21
runtime.systemstack(0x971b25c0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/asm_386.s:337 +0x5e
runtime.mstart()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/proc.go:1132

goroutine 3 [running]:
runtime.systemstack_switch()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/asm_386.s:291 fp=0x96feafbc sp=0x96feafb8
runtime.gosweepone(0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/mgcsweep.go:126 +0x43 fp=0x96feafd0 sp=0x96feafbc
runtime.bgsweep(0x96fc2300)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/mgcsweep.go:59 +0xa1 fp=0x96feafe8 sp=0x96feafd0
runtime.goexit()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96feafec sp=0x96feafe8
created by runtime.gcenable
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/mgc.go:212 +0x55

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x97183dbc)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/sema.go:61 +0x29
sync.(*WaitGroup).Wait(0x97183db0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/sync/waitgroup.go:131 +0x8d
main.cmdDaemon(0x0, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main_daemon.go:99 +0x295
main.run(0x0, 0x1)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main.go:243 +0x34d
main.main()
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main.go:51 +0x1e

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/asm_386.s:1615 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/sigqueue.go:116 +0x14b
os/signal.loop()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/os/signal/signal_unix.go:22 +0x1a
created by os/signal.init.1
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [chan receive]:
database/sql.(*DB).connectionOpener(0x970a2060)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/database/sql/sql.go:837 +0x3a
created by database/sql.Open
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/database/sql/sql.go:582 +0x1c4

goroutine 84 [IO wait]:
net.runtime_pollWait(0x855102a8, 0x72, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/netpoll.go:164 +0x4b
net.(*pollDesc).wait(0x970c52fc, 0x72, 0x0, 0x9711e8c0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:75 +0x2b
net.(*pollDesc).waitRead(0x970c52fc, 0xffffffff, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:80 +0x29
net.(*netFD).accept(0x970c52c0, 0x0, 0x883e400, 0x9711e8c0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_unix.go:430 +0x156
net.(*UnixListener).accept(0x970f08a0, 0x8165796, 0x859e0a0, 0x9711cae0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/unixsock_posix.go:162 +0x27
net.(*UnixListener).Accept(0x970f08a0, 0x9711cac0, 0x855fd20, 0x8834a00, 0x85a86e0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/unixsock.go:237 +0x3c
net/http.(*Server).Serve(0x9719a180, 0x8840dc0, 0x970f08a0, 0x0, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:2647 +0x1a1
main.(*Daemon).Init.func5.3(0x0, 0x807a23e)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:1037 +0x45
gopkg.in/tomb%2ev2.(*Tomb).run(0x9703e2bc, 0x971a8248)
	/lxd/build/tmp.caGjJTJWBE/go/src/gopkg.in/tomb.v2/tomb.go:163 +0x21
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/lxd/build/tmp.caGjJTJWBE/go/src/gopkg.in/tomb.v2/tomb.go:159 +0xae

goroutine 83 [IO wait]:
net.runtime_pollWait(0x85510230, 0x72, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/netpoll.go:164 +0x4b
net.(*pollDesc).wait(0x97187d3c, 0x72, 0x0, 0x97063780)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:75 +0x2b
net.(*pollDesc).waitRead(0x97187d3c, 0xffffffff, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:80 +0x29
net.(*netFD).accept(0x97187d00, 0x0, 0x883e400, 0x97063780)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_unix.go:430 +0x156
net.(*UnixListener).accept(0x97190cc0, 0x970f8740, 0x8097a80, 0x970f873c)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/unixsock_posix.go:162 +0x27
net.(*UnixListener).Accept(0x97190cc0, 0x85ff934, 0x96fde960, 0x88417a0, 0x9711c420)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/unixsock.go:237 +0x3c
net/http.(*Server).Serve(0x9719a100, 0x8840dc0, 0x97190cc0, 0x0, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:2647 +0x1a1
net/http.Serve(0x8840dc0, 0x97190cc0, 0x883de20, 0x971a8270, 0x0, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:2311 +0x5e
main.(*Daemon).Init.func5.1(0x0, 0x807a23e)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:1027 +0x7d
gopkg.in/tomb%2ev2.(*Tomb).run(0x9703e2bc, 0x971a8240)
	/lxd/build/tmp.caGjJTJWBE/go/src/gopkg.in/tomb.v2/tomb.go:163 +0x21
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/lxd/build/tmp.caGjJTJWBE/go/src/gopkg.in/tomb.v2/tomb.go:159 +0xae

goroutine 82 [syscall]:
syscall.Syscall(0x3, 0x8, 0x97120000, 0x1000, 0x1, 0x9701e280, 0x86d6e960)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/syscall/asm_linux_386.s:20 +0x5
syscall.read(0x8, 0x97120000, 0x1000, 0x1000, 0x854ff80, 0x1, 0x97120000)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/syscall/zsyscall_linux_386.go:756 +0x45
syscall.Read(0x8, 0x97120000, 0x1000, 0x1000, 0x1000, 0x1000, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/syscall/syscall_unix.go:162 +0x39
main.deviceNetlinkListener.func1(0x800, 0x8, 0x970e2140, 0x970e2180, 0x970e21c0)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/devices.go:339 +0x89
created by main.deviceNetlinkListener
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/devices.go:336 +0x198

goroutine 165 [runnable]:
golang.org/x/crypto/scrypt.blockMix(0x971a59c0, 0x97218800, 0x200, 0x200, 0x97218c00, 0x100, 0x100, 0x8)
	/lxd/build/tmp.caGjJTJWBE/go/src/golang.org/x/crypto/scrypt/scrypt.go:162 +0x1e7
golang.org/x/crypto/scrypt.smix(0x97094000, 0x400, 0x400, 0x8, 0x4000, 0x97238000, 0x400000, 0x400000, 0x97218800, 0x200, ...)
	/lxd/build/tmp.caGjJTJWBE/go/src/golang.org/x/crypto/scrypt/scrypt.go:187 +0x205
golang.org/x/crypto/scrypt.Key(0x97063c98, 0x3, 0x8, 0x97084620, 0x20, 0x20, 0x4000, 0x8, 0x1, 0x40, ...)
	/lxd/build/tmp.caGjJTJWBE/go/src/golang.org/x/crypto/scrypt/scrypt.go:239 +0x236
main.daemonConfigSetPassword(0x9703e280, 0x85d3e90, 0x13, 0x97063bd0, 0x3, 0x0, 0x971a5bb8, 0x971a5bb4, 0x971a5bb0)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon_config.go:257 +0xff
main.(*daemonConfigKey).Set(0x970c4d00, 0x9703e280, 0x97063bd0, 0x3, 0x97041464, 0x97084401)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon_config.go:112 +0x1c9
main.doApi10Update(0x9703e280, 0x97084280, 0x970843a0, 0x96fbca50, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/api_1.0.go:336 +0x656
main.api10Put(0x9703e280, 0x9719a380, 0x85c9ff7, 0x3)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/api_1.0.go:263 +0x124
main.(*Daemon).createCmd.func1(0x8840e80, 0x96fc8120, 0x9719a380)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:314 +0x543
net/http.HandlerFunc.ServeHTTP(0x971557d0, 0x8840e80, 0x96fc8120, 0x9719a380)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:1942 +0x34
github.com/gorilla/mux.(*Router).ServeHTTP(0x970f3350, 0x8840e80, 0x96fc8120, 0x9719a380)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/gorilla/mux/mux.go:114 +0xce
main.(*lxdHttpServer).ServeHTTP(0x971a8270, 0x8840e80, 0x96fc8120, 0x9719a280)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:1381 +0x1cb
net/http.serverHandler.ServeHTTP(0x9719a100, 0x8840e80, 0x96fc8120, 0x9719a280)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:2572 +0x70
net/http.(*conn).serve(0x96fde960, 0x88416e0, 0x96fbf3a0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:1825 +0x580
created by net/http.(*Server).Serve
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:2672 +0x225

goroutine 168 [IO wait]:
net.runtime_pollWait(0x85510140, 0x72, 0x970f203d)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/netpoll.go:164 +0x4b
net.(*pollDesc).wait(0x9708a07c, 0x72, 0x883efc0, 0x883c924)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:75 +0x2b
net.(*pollDesc).waitRead(0x9708a07c, 0x970f203d, 0x1)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:80 +0x29
net.(*netFD).Read(0x9708a040, 0x970f203d, 0x1, 0x1, 0x0, 0x883efc0, 0x883c924)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_unix.go:250 +0x144
net.(*conn).Read(0x96fbc1c8, 0x970f203d, 0x1, 0x1, 0x0, 0x0, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/net.go:181 +0x56
net/http.(*connReader).backgroundRead(0x970f2030)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:656 +0x48
created by net/http.(*connReader).startBackgroundRead
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:652 +0xa2

goroutine 78 [IO wait]:
net.runtime_pollWait(0x855101b8, 0x72, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/netpoll.go:164 +0x4b
net.(*pollDesc).wait(0x96fc287c, 0x72, 0x0, 0x96fba230)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:75 +0x2b
net.(*pollDesc).waitRead(0x96fc287c, 0xffffffff, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_poll_runtime.go:80 +0x29
net.(*netFD).accept(0x96fc2840, 0x0, 0x883e400, 0x96fba230)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/fd_unix.go:430 +0x156
net.(*TCPListener).accept(0x96fbca90, 0x0, 0x96fe0000, 0x96fc6360)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/tcpsock_posix.go:136 +0x23
net.(*TCPListener).Accept(0x96fbca90, 0x8072701, 0x970960c0, 0x8165796, 0x859e0a0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/tcpsock.go:228 +0x3c
crypto/tls.(*listener).Accept(0x97063c50, 0x970960a0, 0x855fd20, 0x8834a00, 0x85a8560)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/crypto/tls/tls.go:52 +0x2a
net/http.(*Server).Serve(0x96fb8700, 0x8840be0, 0x97063c50, 0x0, 0x0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:2647 +0x1a1
net/http.Serve(0x8840be0, 0x97063c50, 0x883de20, 0x971a8010, 0x970a04c8, 0x970a04c8)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/net/http/server.go:2311 +0x5e
main.(*Daemon).UpdateHTTPsPort.func1(0x96fbc198, 0x807a23e)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:520 +0x8b
gopkg.in/tomb%2ev2.(*Tomb).run(0x9703e2bc, 0x97063c60)
	/lxd/build/tmp.caGjJTJWBE/go/src/gopkg.in/tomb.v2/tomb.go:163 +0x21
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/lxd/build/tmp.caGjJTJWBE/go/src/gopkg.in/tomb.v2/tomb.go:159 +0xae

goroutine 64 [chan receive]:
main.(*Daemon).Init.func2(0x9703e280)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:851 +0xd1
created by main.(*Daemon).Init
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:840 +0xe51

goroutine 65 [select]:
main.deviceEventListener(0x9703e280)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/devices.go:745 +0x51e
created by main.(*Daemon).Init
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:871 +0x1ed4

goroutine 101 [select]:
main.(*Daemon).Ready.func1(0x9703e280)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:1060 +0x105
created by main.(*Daemon).Ready
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:1055 +0x6f

goroutine 102 [select]:
main.(*Daemon).Ready.func2(0x9703e280)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:1088 +0x1e0
created by main.(*Daemon).Ready
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/daemon.go:1074 +0xbf

goroutine 107 [chan receive]:
main.cmdDaemon.func2(0x9703e280, 0x97137400, 0x97183db0)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main_daemon.go:66 +0xb1
created by main.cmdDaemon
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main_daemon.go:63 +0x22d

goroutine 108 [chan receive]:
main.cmdDaemon.func3(0x9703e280, 0x97137400, 0x97183db0)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main_daemon.go:77 +0x3a
created by main.cmdDaemon
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main_daemon.go:76 +0x25b

goroutine 109 [chan receive]:
main.cmdDaemon.func4(0x97137400, 0x9703e280, 0x97183db0)
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main_daemon.go:92 +0x141
created by main.cmdDaemon
	/lxd/build/tmp.caGjJTJWBE/go/src/github.com/lxc/lxd/lxd/main_daemon.go:87 +0x289

goroutine 110 [select, locked to thread]:
runtime.gopark(0x85ffc44, 0x0, 0x85cba57, 0x6, 0x18, 0x2)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/proc.go:271 +0x109
runtime.selectgoImpl(0x97139fa0, 0x0, 0xc)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/select.go:423 +0x10a9
runtime.selectgo(0x97139fa0)
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/select.go:238 +0xf
runtime.ensureSigM.func1()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/signal_unix.go:492 +0x28a
runtime.goexit()
	/lxd/build/tmp.caGjJTJWBE/go/golang/src/runtime/asm_386.s:1615 +0x1
error: Put http://unix.socket/1.0: EOF

@bradfitz bradfitz added this to the Go1.8Maybe milestone Feb 12, 2017

@bradfitz

This comment has been minimized.

Copy link
Member

commented Feb 12, 2017

Ping @aclements, @rsc.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2017

@cynecx, what's the probability of a given 'go get -u -v github.com/nsf/gocode' crashing? I just ran 100 in a row without a problem, using Go 1.7.5 on Linux 3.13.0.

@brauner, are you also using Go 1.7.5? Do you have a command that reproduces the problem reliably? And, unrelated but for my curiosity, why are you using GOARCH=386 instead of amd64?

@aclements

This comment has been minimized.

Copy link
Member

commented Feb 13, 2017

@cynecx and/or @brauner, can you reproduce this with GODEBUG=gccheckmark=1?

@brauner

This comment has been minimized.

Copy link

commented Feb 14, 2017

@rsc, it seems our test for Go 1.7.* ran Go 1.7. We are currently setting up a testrun with Go 1.7.5.
We have no minimal reproducer. Our testsuite manages to produce the increased sweep allocation count error on most test runs on at least one builder; so it's more probability than reliability. As I said, I'm mainly reporting this in case this is a golang regression on Go master. (There is also the possibility that something weird is happening on our side.)
About GOARCH=386. We're testing on multiple architectures including i386.

@aclements, we've set up a testrun with GODEBUG=gccheckmark=1. I will report back once it has finished.

@brauner

This comment has been minimized.

Copy link

commented Feb 14, 2017

@rsc, one more detail, the error seems independent of the arch we test on. It happens with equal probability on i386 and amd64.

@aclements

This comment has been minimized.

Copy link
Member

commented Feb 14, 2017

@brauner, another question: is there any unsafe or cgo in your application? For example, creating a pointer into the Go heap that points at an unallocated object can result in exactly this failure.

@brauner

This comment has been minimized.

Copy link

commented Feb 14, 2017

@aclements, yes we're using cgo and we're using unsafe. So here's another stacktrace from Go master that might be more informative:

=== RUN   TestContainer_ProfilesMulti
runtime:greyobject: checkmarks finds unexpected unmarked object obj=0x972d4a50
runtime: found obj at *(0x970b2fc8+0xc)
base=0x970b2fc8 k=0x4b859 s.base()=0x970ac000 s.limit=0x0 s.sizeclass=0 s.elemsize=0 s.state=_MSpanStack
 *(base+0) = 0x973274f0
 *(base+4) = 0x8864cc0
 *(base+8) = 0x96fba150
 *(base+12) = 0x972d4a50 <==
obj=0x972d4a50 k=0x4b96a s.base()=0x972d4000 s.limit=0x972d5fe0 s.sizeclass=6 s.elemsize=80 s.state=_MSpanInUse
 *(obj+0) = 0x9746e960
 *(obj+4) = 0x0
 *(obj+8) = 0x0
 *(obj+12) = 0x0
 *(obj+16) = 0x0
 *(obj+20) = 0x0
 *(obj+24) = 0x0
 *(obj+28) = 0x0
 *(obj+32) = 0x0
 *(obj+36) = 0x0
 *(obj+40) = 0x1
 *(obj+44) = 0x0
 *(obj+48) = 0x0
 *(obj+52) = 0x972d1b00
 *(obj+56) = 0x5
 *(obj+60) = 0x8
 *(obj+64) = 0x972ccd20
 *(obj+68) = 0x8868d60
 *(obj+72) = 0x972d17e0
 *(obj+76) = 0x0
fatal error: checkmark found unmarked object

runtime stack:
runtime.throw(0x85f08fb, 0x1f)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/panic.go:596 +0x7c fp=0xbf9b72ec sp=0xbf9b72e0
runtime.greyobject(0x972d4a50, 0x970b2fc8, 0xc, 0x8d87eb5a, 0x0, 0x85512610, 0x96fd1e2c, 0x21)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:1384 +0x2bf fp=0xbf9b7300 sp=0xbf9b72ec
runtime.scanblock(0x970b2fc8, 0x14, 0x861b189, 0x96fd1e2c)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:1231 +0x112 fp=0xbf9b7338 sp=0xbf9b7300
runtime.scanframeworker(0xbf9b7500, 0xbf9b7574, 0x96fd1e2c)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:935 +0xad fp=0xbf9b7378 sp=0xbf9b7338
runtime.scanstack.func1(0xbf9b7500, 0x0, 0xbf9b7401)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:829 +0x56 fp=0xbf9b739c sp=0xbf9b7378
runtime.gentraceback(0xffffffff, 0x970b2ea0, 0x0, 0x9709de00, 0x0, 0x0, 0x7fffffff, 0xbf9b7634, 0x0, 0x0, ...)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/traceback.go:378 +0xe32 fp=0xbf9b752c sp=0xbf9b739c
runtime.scanstack(0x9709de00, 0x96fd1e2c)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:852 +0x1fa fp=0xbf9b7658 sp=0xbf9b752c
runtime.scang(0x9709de00, 0x96fd1e2c)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:866 +0x219 fp=0xbf9b7678 sp=0xbf9b7658
runtime.markroot.func1()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:264 +0x5c fp=0xbf9b7698 sp=0xbf9b7678
runtime.systemstack(0xbf9b76d8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:353 +0x88 fp=0xbf9b769c sp=0xbf9b7698
runtime.markroot(0x96fd1e2c, 0x45)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:269 +0x191 fp=0xbf9b76e8 sp=0xbf9b769c
runtime.gcDrain(0x96fd1e2c, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:1061 +0xe0 fp=0xbf9b7718 sp=0xbf9b76e8
runtime.gchelper()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1911 +0x88 fp=0xbf9b7734 sp=0xbf9b7718
runtime.stopm()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:1653 +0xc3 fp=0xbf9b7744 sp=0xbf9b7734
runtime.findrunnable(0x96fce000, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:2102 +0x431 fp=0xbf9b77a0 sp=0xbf9b7744
runtime.schedule()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:2222 +0x103 fp=0xbf9b77bc sp=0xbf9b77a0
runtime.park_m(0x9709ce10)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:2285 +0x95 fp=0xbf9b77d4 sp=0xbf9b77bc
runtime.mcall(0xbf9b77e8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:279 +0x47 fp=0xbf9b77dc sp=0xbf9b77d4

goroutine 1 [chan receive]:
runtime.gopark(0x8611c88, 0x97149a70, 0x85e2706, 0xc, 0x974ac617, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x974c0d9c sp=0x974c0d88
runtime.goparkunlock(0x97149a70, 0x85e2706, 0xc, 0x17, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x974c0db8 sp=0x974c0d9c
runtime.chanrecv(0x853fb80, 0x97149a40, 0x0, 0x974c0e01, 0x8116b31)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:513 +0x2ee fp=0x974c0e08 sp=0x974c0db8
runtime.chanrecv1(0x853fb80, 0x97149a40, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:395 +0x24 fp=0x974c0e20 sp=0x974c0e08
testing.(*T).Run(0x96fc70e0, 0x85e47b7, 0x10, 0x8611204, 0x8116801)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:702 +0x282 fp=0x974c0e70 sp=0x974c0e20
testing.runTests.func1(0x96fc70e0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:888 +0x4d fp=0x974c0e94 sp=0x974c0e70
testing.tRunner(0x96fc70e0, 0x96ffbf04)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:659 +0x79 fp=0x974c0ec0 sp=0x974c0e94
testing.runTests(0x9707cf30, 0x8860380, 0x15, 0x15, 0x96fcb880)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:894 +0x23e fp=0x974c0f18 sp=0x974c0ec0
testing.(*M).Run(0x974c0f98, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:828 +0xc0 fp=0x974c0f7c sp=0x974c0f18
main.main()
	github.com/lxc/lxd/lxd/_test/_testmain.go:86 +0xc6 fp=0x974c0fc8 sp=0x974c0f7c
runtime.main()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:185 +0x1f2 fp=0x974c0ff0 sp=0x974c0fc8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x974c0ff4 sp=0x974c0ff0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96feaff4 sp=0x96feaff0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x8611c88, 0x888e298, 0x85e40af, 0xf, 0x814, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x96fd87bc sp=0x96fd87a8
runtime.goparkunlock(0x888e298, 0x85e40af, 0xf, 0x96fac014, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x96fd87d8 sp=0x96fd87bc
runtime.forcegchelper()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:226 +0x99 fp=0x96fd87f0 sp=0x96fd87d8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96fd87f4 sp=0x96fd87f0
created by runtime.init.3
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:215 +0x2b

goroutine 3 [GC sweep wait]:
runtime.gopark(0x8611c88, 0x888e480, 0x85e2de9, 0xd, 0x8098b14, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x96fd8fb4 sp=0x96fd8fa0
runtime.goparkunlock(0x888e480, 0x85e2de9, 0xd, 0x14, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x96fd8fd0 sp=0x96fd8fb4
runtime.bgsweep(0x96fc4200)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcsweep.go:72 +0x109 fp=0x96fd8fe8 sp=0x96fd8fd0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96fd8fec sp=0x96fd8fe8
created by runtime.gcenable
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:212 +0x55

goroutine 4 [finalizer wait]:
runtime.gopark(0x8611c88, 0x889fd74, 0x85e380b, 0xe, 0x14, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x96fd9794 sp=0x96fd9780
runtime.goparkunlock(0x889fd74, 0x85e380b, 0xe, 0x14, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x96fd97b0 sp=0x96fd9794
runtime.runfinq()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mfinal.go:161 +0x9c fp=0x96fd97f0 sp=0x96fd97b0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96fd97f4 sp=0x96fd97f0
created by runtime.createfing
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mfinal.go:142 +0x60

goroutine 5 [syscall]:
runtime.notetsleepg(0x88a0180, 0xffffffff, 0xffffffff, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/lock_futex.go:205 +0x37 fp=0x96fd9fc0 sp=0x96fd9fa8
os/signal.signal_recv(0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/sigqueue.go:116 +0x14b fp=0x96fd9fd8 sp=0x96fd9fc0
os/signal.loop()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/os/signal/signal_unix.go:22 +0x1a fp=0x96fd9ff0 sp=0x96fd9fd8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96fd9ff4 sp=0x96fd9ff0
created by os/signal.init.1
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/os/signal/signal_unix.go:28 +0x37

goroutine 1272 [chan receive]:
runtime.gopark(0x8611c88, 0x9731cf70, 0x85e2706, 0xc, 0x971f6017, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x96fe6d54 sp=0x96fe6d40
runtime.goparkunlock(0x9731cf70, 0x85e2706, 0xc, 0x117, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x96fe6d70 sp=0x96fe6d54
runtime.chanrecv(0x853fb80, 0x9731cf40, 0x0, 0x96fe6d01, 0x8116b31)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:513 +0x2ee fp=0x96fe6dc0 sp=0x96fe6d70
runtime.chanrecv1(0x853fb80, 0x9731cf40, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:395 +0x24 fp=0x96fe6dd8 sp=0x96fe6dc0
testing.(*T).Run(0x97257440, 0x8518359, 0x1b, 0x9725b800, 0x8116801)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:702 +0x282 fp=0x96fe6e28 sp=0x96fe6dd8
testing.runTests.func1(0x97257440)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:888 +0x4d fp=0x96fe6e4c sp=0x96fe6e28
testing.tRunner(0x97257440, 0x96fe6ebc)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:659 +0x79 fp=0x96fe6e78 sp=0x96fe6e4c
testing.runTests(0x8611588, 0x97266b40, 0xf, 0x10, 0x85dd2f2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:894 +0x23e fp=0x96fe6ed0 sp=0x96fe6e78
testing.RunTests(0x8611588, 0x97266b40, 0xf, 0x10, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:865 +0x3d fp=0x96fe6f00 sp=0x96fe6ed0
github.com/stretchr/testify/suite.Run(0x9747b0e0, 0x88670c0, 0x970cf160)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/stretchr/testify/suite/suite.go:109 +0x480 fp=0x96fe6fac sp=0x96fe6f00
github.com/lxc/lxd/lxd.TestLxdTestSuite(0x9747b0e0)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/main_test.go:76 +0x41 fp=0x96fe6fbc sp=0x96fe6fac
testing.tRunner(0x9747b0e0, 0x8611204)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:659 +0x79 fp=0x96fe6fe8 sp=0x96fe6fbc
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96fe6fec sp=0x96fe6fe8
created by testing.(*T).Run
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:701 +0x261

goroutine 2453 [runnable, locked to thread]:
runtime.cgocall(0x84f2160, 0x973dcb10, 0x8068136)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/cgocall.go:132 +0xe8 fp=0x973dcafc sp=0x973dcae0
github.com/mattn/go-sqlite3._Cfunc_sqlite3_step(0x80e562d8, 0x65)
	github.com/mattn/go-sqlite3/_obj/_cgo_gotypes.go:831 +0x3b fp=0x973dcb10 sp=0x973dcafc
github.com/mattn/go-sqlite3.(*SQLiteRows).Next.func1(0x80e562d8, 0x9732aebc)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/mattn/go-sqlite3/sqlite3.go:878 +0x4f fp=0x973dcb28 sp=0x973dcb10
github.com/mattn/go-sqlite3.(*SQLiteRows).Next(0x97321530, 0x9732aee0, 0x3, 0x3, 0x3, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/mattn/go-sqlite3/sqlite3.go:878 +0x45 fp=0x973dccfc sp=0x973dcb28
database/sql.(*Rows).nextLocked(0x97327590, 0x973dcd30)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2149 +0x57 fp=0x973dcd18 sp=0x973dccfc
database/sql.(*Rows).Next.func1()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2134 +0x2e fp=0x973dcd2c sp=0x973dcd18
database/sql.withLock(0x8867b40, 0x973275a8, 0x973dcd4c)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2545 +0x52 fp=0x973dcd3c sp=0x973dcd2c
database/sql.(*Rows).Next(0x97327590, 0x889fdf8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2135 +0x68 fp=0x973dcd60 sp=0x973dcd3c
github.com/lxc/lxd/lxd.doDbQueryScan(0x9746e960, 0x860745f, 0xba, 0x97328818, 0x1, 0x1, 0x973dcefc, 0x3, 0x3, 0x0, ...)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/db.go:376 +0x109 fp=0x973dcde4 sp=0x973dcd60
github.com/lxc/lxd/lxd.dbQueryScan(0x9746e960, 0x860745f, 0xba, 0x97328818, 0x1, 0x1, 0x973dcefc, 0x3, 0x3, 0x20, ...)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/db.go:433 +0x98 fp=0x973dce6c sp=0x973dcde4
github.com/lxc/lxd/lxd.dbDevices(0x9746e960, 0x85dea80, 0x7, 0x9732ae01, 0x0, 0x9732ae20, 0x805b999)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/db_devices.go:150 +0x1bd fp=0x973dcf18 sp=0x973dce6c
github.com/lxc/lxd/lxd.(*containerLXC).expandDevices(0x972f6d80, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/container_lxc.go:1407 +0x99 fp=0x973dcfc4 sp=0x973dcf18
github.com/lxc/lxd/lxd.(*containerLXC).Update(0x972f6d80, 0x0, 0x1, 0x0, 0x0, 0x9731bfe0, 0x0, 0x0, 0x0, 0x0, ...)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/container_lxc.go:3023 +0xc62 fp=0x973dd9a4 sp=0x973dcfc4
github.com/lxc/lxd/lxd.containerLXCCreate(0x97362000, 0x8, 0x1, 0x0, 0x0, 0x9731bfe0, 0x0, 0x0, 0xd034542c, 0xe, ...)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/container_lxc.go:246 +0x797 fp=0x973ddae0 sp=0x973dd9a4
github.com/lxc/lxd/lxd.containerCreateInternal(0x97362000, 0x8, 0x1, 0x0, 0x0, 0x9731bfe0, 0x0, 0x0, 0xd034542c, 0xe, ...)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/container.go:674 +0x5bb fp=0x973ddc7c sp=0x973ddae0
github.com/lxc/lxd/lxd.(*lxdTestSuite).TestContainer_ProfilesMulti(0x970cf160)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/container_test.go:55 +0x29d fp=0x973ddd90 sp=0x973ddc7c
runtime.call16(0x97255680, 0x97272310, 0x97272620, 0x4, 0x4)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:519 +0x3a fp=0x973ddda4 sp=0x973ddd90
reflect.Value.call(0x97504780, 0x97272310, 0x13, 0x85dcb24, 0x4, 0x972b8fac, 0x1, 0x1, 0x972b8f4c, 0x85d9f80, ...)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/reflect/value.go:434 +0x9fe fp=0x973ddf18 sp=0x973ddda4
reflect.Value.Call(0x97504780, 0x97272310, 0x13, 0x972b8fac, 0x1, 0x1, 0x8518359, 0x1b, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/reflect/value.go:302 +0x7f fp=0x973ddf48 sp=0x973ddf18
github.com/stretchr/testify/suite.Run.func2(0x973190e0)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/stretchr/testify/suite/suite.go:101 +0x1a5 fp=0x973ddfbc sp=0x973ddf48
testing.tRunner(0x973190e0, 0x9725b800)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:659 +0x79 fp=0x973ddfe8 sp=0x973ddfbc
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x973ddfec sp=0x973ddfe8
created by testing.(*T).Run
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/testing/testing.go:701 +0x261

goroutine 1319 [chan receive]:
runtime.gopark(0x8611c88, 0x9704b6f0, 0x85e2706, 0xc, 0x808ae17, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970f072c sp=0x970f0718
runtime.goparkunlock(0x9704b6f0, 0x85e2706, 0xc, 0x885be17, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x970f0748 sp=0x970f072c
runtime.chanrecv(0x853fa80, 0x9704b6c0, 0x0, 0x1, 0x17)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:513 +0x2ee fp=0x970f0798 sp=0x970f0748
runtime.chanrecv1(0x853fa80, 0x9704b6c0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:395 +0x24 fp=0x970f07b0 sp=0x970f0798
github.com/lxc/lxd/lxd.(*Daemon).Init.func2(0x97362000)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:839 +0xd1 fp=0x970f07e8 sp=0x970f07b0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970f07ec sp=0x970f07e8
created by github.com/lxc/lxd/lxd.(*Daemon).Init
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:828 +0xd34

goroutine 2495 [select]:
runtime.gopark(0x8611ca4, 0x0, 0x85de274, 0x6, 0x18, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970a85f8 sp=0x970a85e4
runtime.selectgoImpl(0x970a8788, 0x0, 0xc)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:423 +0x10a0 fp=0x970a8760 sp=0x970a85f8
runtime.selectgo(0x970a8788)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:238 +0xf fp=0x970a8770 sp=0x970a8760
github.com/mattn/go-sqlite3.(*SQLiteStmt).query.func2(0x8550c1b8, 0x96fb8200, 0x97321530, 0x9732ae80, 0x80201088)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/mattn/go-sqlite3/sqlite3.go:770 +0x10f fp=0x970a87d8 sp=0x970a8770
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970a87dc sp=0x970a87d8
created by github.com/mattn/go-sqlite3.(*SQLiteStmt).query
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/mattn/go-sqlite3/sqlite3.go:769 +0x158

goroutine 2462 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970b16b4 sp=0x970b16a0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x970b16d0 sp=0x970b16b4
runtime.gcParkAssist(0x9709db88)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x970b16f0 sp=0x970b16d0
runtime.gcAssistAlloc(0x9709db30)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x970b1748 sp=0x970b16f0
runtime.mallocgc(0x8, 0x8571ca0, 0x9709db01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x970b1798 sp=0x970b1748
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x970b17ac sp=0x970b1798
database/sql.(*Rows).close(0x97313bd0, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x970b17c8 sp=0x970b17ac
database/sql.(*Rows).awaitDone(0x97313bd0, 0x8868d60, 0x97322360)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x970b17e0 sp=0x970b17c8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970b17e4 sp=0x970b17e0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 2492 [runnable (scan)]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970b2eb4 sp=0x970b2ea0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x970b2ed0 sp=0x970b2eb4
runtime.gcParkAssist(0x9709de58)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x970b2ef0 sp=0x970b2ed0
runtime.gcAssistAlloc(0x9709de00)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x970b2f48 sp=0x970b2ef0
runtime.mallocgc(0x8, 0x8571ca0, 0x9709de01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x970b2f98 sp=0x970b2f48
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x970b2fac sp=0x970b2f98
database/sql.(*Rows).close(0x973274f0, 0x8864cc0, 0x96fba150, 0x972d4a50, 0x972cec01)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x970b2fc8 sp=0x970b2fac
database/sql.(*Rows).awaitDone(0x973274f0, 0x8868d60, 0x9732ace0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x970b2fe0 sp=0x970b2fc8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970b2fe4 sp=0x970b2fe0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 1266 [IO wait]:
runtime.gopark(0x8611c6c, 0x855111dc, 0x85de6f9, 0x7, 0x80ab91b, 0x5)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x974beacc sp=0x974beab8
runtime.netpollblock(0x855111c8, 0x72, 0x0, 0xb)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:345 +0xb2 fp=0x974beaec sp=0x974beacc
internal/poll.runtime_pollWait(0x855111c8, 0x72, 0xa)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:164 +0x4b fp=0x974beb00 sp=0x974beaec
internal/poll.(*pollDesc).wait(0x9746a014, 0x72, 0x8866400, 0x8863d40)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:84 +0x8c fp=0x974beb14 sp=0x974beb00
internal/poll.(*pollDesc).waitRead(0x9746a014, 0x97472000, 0x1000)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:89 +0x29 fp=0x974beb28 sp=0x974beb14
internal/poll.(*FD).Read(0x9746a000, 0x97472000, 0x1000, 0x1000, 0x0, 0x8866400, 0x8863d40)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_unix.go:113 +0x140 fp=0x974beb4c sp=0x974beb28
net.(*netFD).Read(0x9746a000, 0x97472000, 0x1000, 0x1000, 0x9746c064, 0xffffffff, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/fd_unix.go:207 +0x3f fp=0x974beb7c sp=0x974beb4c
net.(*conn).Read(0x9710c008, 0x97472000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/net.go:182 +0x56 fp=0x974bebac sp=0x974beb7c
net/http.(*connReader).Read(0x9746c060, 0x97472000, 0x1000, 0x1000, 0xfceb8072, 0x6, 0x888e6e0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/server.go:754 +0x13b fp=0x974bebd8 sp=0x974bebac
bufio.(*Reader).fill(0x9746c090)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/bufio/bufio.go:97 +0xdf fp=0x974bec00 sp=0x974bebd8
bufio.(*Reader).ReadSlice(0x9746c090, 0xa, 0x805975e, 0x805a5df, 0xffffffff, 0x1fffff, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/bufio/bufio.go:338 +0x90 fp=0x974bec2c sp=0x974bec00
bufio.(*Reader).ReadLine(0x9746c090, 0x80, 0x80, 0x85ca3e0, 0x81b6d63, 0x900, 0x80)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/bufio/bufio.go:367 +0x2a fp=0x974bec60 sp=0x974bec2c
net/textproto.(*Reader).readLineSlice(0x970ce140, 0x80000000, 0x8287797, 0x805b1ec, 0x80, 0x85ca3e0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/textproto/reader.go:55 +0x45 fp=0x974beca0 sp=0x974bec60
net/textproto.(*Reader).ReadLine(0x970ce140, 0x9701a580, 0x9746a000, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/textproto/reader.go:36 +0x21 fp=0x974becc4 sp=0x974beca0
net/http.readRequest(0x9746c090, 0x0, 0x9701a580, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/request.go:918 +0x6f fp=0x974bed24 sp=0x974becc4
net/http.(*conn).readRequest(0x9746e000, 0x8868d60, 0x970ce0a0, 0x0, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/server.go:934 +0x1b8 fp=0x974bee00 sp=0x974bed24
net/http.(*conn).serve(0x9746e000, 0x8868d60, 0x970ce0a0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/server.go:1763 +0x405 fp=0x974befe0 sp=0x974bee00
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x974befe4 sp=0x974befe0
created by net/http.(*Server).Serve
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/server.go:2672 +0x225

goroutine 1199 [select]:
runtime.gopark(0x8611ca4, 0x0, 0x85de274, 0x6, 0x18, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970e54f4 sp=0x970e54e0
runtime.selectgoImpl(0x970e5758, 0x0, 0xc)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:423 +0x10a0 fp=0x970e565c sp=0x970e54f4
runtime.selectgo(0x970e5758)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:238 +0xf fp=0x970e566c sp=0x970e565c
github.com/lxc/lxd/lxd.deviceEventListener(0x9705ab40)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/devices.go:745 +0x51e fp=0x970e57e8 sp=0x970e566c
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970e57ec sp=0x970e57e8
created by github.com/lxc/lxd/lxd.(*Daemon).Init
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:859 +0x1dc9

goroutine 2494 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970f3eb4 sp=0x970f3ea0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x970f3ed0 sp=0x970f3eb4
runtime.gcParkAssist(0x970ecaa8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x970f3ef0 sp=0x970f3ed0
runtime.gcAssistAlloc(0x970eca50)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x970f3f48 sp=0x970f3ef0
runtime.mallocgc(0x8, 0x8571ca0, 0x970eca01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x970f3f98 sp=0x970f3f48
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x970f3fac sp=0x970f3f98
database/sql.(*Rows).close(0x97327540, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x970f3fc8 sp=0x970f3fac
database/sql.(*Rows).awaitDone(0x97327540, 0x8868d60, 0x9732ad80)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x970f3fe0 sp=0x970f3fc8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970f3fe4 sp=0x970f3fe0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 2496 [chan receive]:
runtime.gopark(0x8611c88, 0x9732c0b0, 0x85e2706, 0xc, 0x970f1717, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970f1744 sp=0x970f1730
runtime.goparkunlock(0x9732c0b0, 0x85e2706, 0xc, 0x805ba17, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x970f1760 sp=0x970f1744
runtime.chanrecv(0x853fa40, 0x9732c080, 0x0, 0x9732c001, 0x9732aea8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:513 +0x2ee fp=0x970f17b0 sp=0x970f1760
runtime.chanrecv1(0x853fa40, 0x9732c080, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:395 +0x24 fp=0x970f17c8 sp=0x970f17b0
database/sql.(*Rows).awaitDone(0x97327590, 0x8868d60, 0x9732aea0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2121 +0x43 fp=0x970f17e0 sp=0x970f17c8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970f17e4 sp=0x970f17e0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 2478 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x970ffeb4 sp=0x970ffea0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x970ffed0 sp=0x970ffeb4
runtime.gcParkAssist(0x970edc78)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x970ffef0 sp=0x970ffed0
runtime.gcAssistAlloc(0x970edc20)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x970fff48 sp=0x970ffef0
runtime.mallocgc(0x8, 0x8571ca0, 0x970edc01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x970fff98 sp=0x970fff48
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x970fffac sp=0x970fff98
database/sql.(*Rows).close(0x97313e00, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x970fffc8 sp=0x970fffac
database/sql.(*Rows).awaitDone(0x97313e00, 0x8868d60, 0x97322960)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x970fffe0 sp=0x970fffc8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x970fffe4 sp=0x970fffe0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 249 [chan receive]:
runtime.gopark(0x8611c88, 0x97111bf0, 0x85e2706, 0xc, 0x8082117, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x97105f4c sp=0x97105f38
runtime.goparkunlock(0x97111bf0, 0x85e2706, 0xc, 0x17, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x97105f68 sp=0x97105f4c
runtime.chanrecv(0x8540240, 0x97111bc0, 0x97105fe0, 0x1, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:513 +0x2ee fp=0x97105fb8 sp=0x97105f68
runtime.chanrecv2(0x8540240, 0x97111bc0, 0x97105fe0, 0x809c626)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:400 +0x24 fp=0x97105fd0 sp=0x97105fb8
database/sql.(*DB).connectionOpener(0x970aaba0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:837 +0x3a fp=0x97105fe8 sp=0x97105fd0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x97105fec sp=0x97105fe8
created by database/sql.Open
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:582 +0x1c4

goroutine 1218 [garbage collection]:
runtime.systemstack_switch()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:291 fp=0x97119dc8 sp=0x97119dc4
runtime.gcMarkTermination()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1277 +0x19c fp=0x97119f88 sp=0x97119dc8
runtime.gcMarkDone()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1207 +0x213 fp=0x97119fa0 sp=0x97119f88
runtime.gcBgMarkWorker(0x96fcf500)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1582 +0x299 fp=0x97119fe8 sp=0x97119fa0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x97119fec sp=0x97119fe8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 1261 [select]:
runtime.gopark(0x8611ca4, 0x0, 0x85de274, 0x6, 0x18, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x9711f5bc sp=0x9711f5a8
runtime.selectgoImpl(0x9711f798, 0x0, 0xc)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:423 +0x10a0 fp=0x9711f724 sp=0x9711f5bc
runtime.selectgo(0x9711f798)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:238 +0xf fp=0x9711f734 sp=0x9711f724
net/http.(*persistConn).writeLoop(0x9705b040)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/transport.go:1704 +0x37e fp=0x9711f7e8 sp=0x9711f734
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x9711f7ec sp=0x9711f7e8
created by net/http.(*Transport).dialConn
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/transport.go:1118 +0x870

goroutine 1260 [IO wait]:
runtime.gopark(0x8611c6c, 0x85511164, 0x85de6f9, 0x7, 0x80ab91b, 0x5)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x974bfc28 sp=0x974bfc14
runtime.netpollblock(0x85511150, 0x72, 0x0, 0xb)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:345 +0xb2 fp=0x974bfc48 sp=0x974bfc28
internal/poll.runtime_pollWait(0x85511150, 0x72, 0x9)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:164 +0x4b fp=0x974bfc5c sp=0x974bfc48
internal/poll.(*pollDesc).wait(0x9726efb4, 0x72, 0x8866400, 0x8863d40)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:84 +0x8c fp=0x974bfc70 sp=0x974bfc5c
internal/poll.(*pollDesc).waitRead(0x9726efb4, 0x97467000, 0x1000)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:89 +0x29 fp=0x974bfc84 sp=0x974bfc70
internal/poll.(*FD).Read(0x9726efa0, 0x97467000, 0x1000, 0x1000, 0x0, 0x8866400, 0x8863d40)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_unix.go:113 +0x140 fp=0x974bfca8 sp=0x974bfc84
net.(*netFD).Read(0x9726efa0, 0x97467000, 0x1000, 0x1000, 0x0, 0x85e1633, 0x80598ef)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/fd_unix.go:207 +0x3f fp=0x974bfcd8 sp=0x974bfca8
net.(*conn).Read(0x96fbbe68, 0x97467000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/net.go:182 +0x56 fp=0x974bfd08 sp=0x974bfcd8
net/http.(*persistConn).Read(0x9705b040, 0x97467000, 0x1000, 0x1000, 0x9728bd50, 0x9704b5f0, 0x8097d70)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/transport.go:1316 +0x15b fp=0x974bfd44 sp=0x974bfd08
bufio.(*Reader).fill(0x972aaf00)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/bufio/bufio.go:97 +0xdf fp=0x974bfd6c sp=0x974bfd44
bufio.(*Reader).Peek(0x972aaf00, 0x1, 0x974bfdad, 0x1, 0x0, 0x97289480, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/bufio/bufio.go:129 +0x51 fp=0x974bfd84 sp=0x974bfd6c
net/http.(*persistConn).readLoop(0x9705b040)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/transport.go:1474 +0x185 fp=0x974bffe8 sp=0x974bfd84
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x974bffec sp=0x974bffe8
created by net/http.(*Transport).dialConn
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/transport.go:1117 +0x84f

goroutine 1254 [select]:
runtime.gopark(0x8611ca4, 0x0, 0x85de274, 0x6, 0x18, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x96fe8e00 sp=0x96fe8dec
runtime.selectgoImpl(0x96fe8f98, 0x0, 0xc)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:423 +0x10a0 fp=0x96fe8f68 sp=0x96fe8e00
runtime.selectgo(0x96fe8f98)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:238 +0xf fp=0x96fe8f78 sp=0x96fe8f68
github.com/lxc/lxd/lxd.(*Daemon).Ready.func2(0x9705ab40)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:1076 +0x1e0 fp=0x96fe8fe8 sp=0x96fe8f78
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96fe8fec sp=0x96fe8fe8
created by github.com/lxc/lxd/lxd.(*Daemon).Ready
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:1062 +0xbf

goroutine 1253 [select]:
runtime.gopark(0x8611ca4, 0x0, 0x85de274, 0x6, 0x18, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x96fe7e04 sp=0x96fe7df0
runtime.selectgoImpl(0x96fe7f98, 0x0, 0xc)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:423 +0x10a0 fp=0x96fe7f6c sp=0x96fe7e04
runtime.selectgo(0x96fe7f98)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/select.go:238 +0xf fp=0x96fe7f7c sp=0x96fe7f6c
github.com/lxc/lxd/lxd.(*Daemon).Ready.func1(0x9705ab40)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:1048 +0x105 fp=0x96fe7fe8 sp=0x96fe7f7c
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x96fe7fec sp=0x96fe7fe8
created by github.com/lxc/lxd/lxd.(*Daemon).Ready
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:1043 +0x6f

goroutine 1206 [IO wait]:
runtime.gopark(0x8611c6c, 0x855112cc, 0x85de6f9, 0x7, 0x1b, 0x5)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x97128e10 sp=0x97128dfc
runtime.netpollblock(0x855112b8, 0x72, 0x0, 0x80800)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:345 +0xb2 fp=0x97128e30 sp=0x97128e10
internal/poll.runtime_pollWait(0x855112b8, 0x72, 0x8866400)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:164 +0x4b fp=0x97128e44 sp=0x97128e30
internal/poll.(*pollDesc).wait(0x9723c9c4, 0x72, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:84 +0x8c fp=0x97128e58 sp=0x97128e44
internal/poll.(*pollDesc).waitRead(0x9723c9c4, 0xffffffff, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:89 +0x29 fp=0x97128e6c sp=0x97128e58
internal/poll.(*FD).Accept(0x9723c9b0, 0x0, 0x0, 0x0, 0x85de888, 0x7, 0x8866400, 0x9725271c)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_unix.go:349 +0x181 fp=0x97128e94 sp=0x97128e6c
net.(*netFD).accept(0x9723c9b0, 0x0, 0x829413d, 0x805b1ec)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/fd_unix.go:243 +0x27 fp=0x97128efc sp=0x97128e94
net.(*UnixListener).accept(0x9723b9a0, 0x81369b6, 0x85ae1c0, 0x97254820)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/unixsock_posix.go:162 +0x27 fp=0x97128f18 sp=0x97128efc
net.(*UnixListener).Accept(0x9723b9a0, 0x97254800, 0x856d700, 0x885b9f8, 0x85b9160)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/unixsock.go:237 +0x3c fp=0x97128f38 sp=0x97128f18
net/http.(*Server).Serve(0x96fb6700, 0x88683e0, 0x9723b9a0, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/server.go:2647 +0x1a1 fp=0x97128f90 sp=0x97128f38
net/http.Serve(0x88683e0, 0x9723b9a0, 0x8864e00, 0x97272120, 0x9713212c, 0x97132128)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/server.go:2311 +0x5e fp=0x97128fac sp=0x97128f90
github.com/lxc/lxd/lxd.(*Daemon).Init.func5.1(0x96fba150, 0x807aaee)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:1015 +0x80 fp=0x97128fd0 sp=0x97128fac
gopkg.in/tomb%2ev2.(*Tomb).run(0x9736203c, 0x97272008)
	/lxd/build/tmp.qYqXiG8rio/go/src/gopkg.in/tomb.v2/tomb.go:163 +0x21 fp=0x97128fe8 sp=0x97128fd0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x97128fec sp=0x97128fe8
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/lxd/build/tmp.qYqXiG8rio/go/src/gopkg.in/tomb.v2/tomb.go:159 +0xae

goroutine 1205 [syscall]:
syscall.Syscall(0x3, 0x7, 0x972b1000, 0x1000, 0x8, 0x8057bd3, 0x9749a260)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/syscall/asm_linux_386.s:20 +0x5 fp=0x9712a5fc sp=0x9712a5f8
syscall.read(0x7, 0x972b1000, 0x1000, 0x1000, 0x9, 0x96fc6900, 0x96fc6900)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/syscall/zsyscall_linux_386.go:756 +0x45 fp=0x9712a624 sp=0x9712a5fc
syscall.Read(0x7, 0x972b1000, 0x1000, 0x1000, 0x96fc6958, 0x0, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/syscall/syscall_unix.go:162 +0x39 fp=0x9712a644 sp=0x9712a624
github.com/lxc/lxd/lxd.deviceNetlinkListener.func1(0x800, 0x7, 0x970d8400, 0x970d8440, 0x970d8480)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/devices.go:339 +0x89 fp=0x9712a7d8 sp=0x9712a644
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x9712a7dc sp=0x9712a7d8
created by github.com/lxc/lxd/lxd.deviceNetlinkListener
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/devices.go:336 +0x198

goroutine 1234 [GC worker (idle)]:
runtime.gopark(0x8611bc8, 0x9728b978, 0x85e4627, 0x10, 0x14, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x9712afa0 sp=0x9712af8c
runtime.gcBgMarkWorker(0x96fd2000)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1492 +0xf9 fp=0x9712afe8 sp=0x9712afa0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x9712afec sp=0x9712afe8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 1219 [GC worker (idle)]:
runtime.gopark(0x8611bc8, 0x9749a008, 0x85e4627, 0x10, 0x14, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x9712bfa0 sp=0x9712bf8c
runtime.gcBgMarkWorker(0x96fd0000)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1492 +0xf9 fp=0x9712bfe8 sp=0x9712bfa0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x9712bfec sp=0x9712bfe8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 1220 [GC worker (idle)]:
runtime.gopark(0x8611bc8, 0x9749a010, 0x85e4627, 0x10, 0x14, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x9716e7a0 sp=0x9716e78c
runtime.gcBgMarkWorker(0x96fd0a80)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1492 +0xf9 fp=0x9716e7e8 sp=0x9716e7a0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x9716e7ec sp=0x9716e7e8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 1221 [GC worker (idle)]:
runtime.gopark(0x8611bc8, 0x9749a018, 0x85e4627, 0x10, 0xffffff14, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x9716efa0 sp=0x9716ef8c
runtime.gcBgMarkWorker(0x96fd1500)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1492 +0xf9 fp=0x9716efe8 sp=0x9716efa0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x9716efec sp=0x9716efe8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 1207 [IO wait]:
runtime.gopark(0x8611c6c, 0x85511254, 0x85de6f9, 0x7, 0x1b, 0x5)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x97264e34 sp=0x97264e20
runtime.netpollblock(0x85511240, 0x72, 0x0, 0x80800)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:345 +0xb2 fp=0x97264e54 sp=0x97264e34
internal/poll.runtime_pollWait(0x85511240, 0x72, 0x8866400)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/netpoll.go:164 +0x4b fp=0x97264e68 sp=0x97264e54
internal/poll.(*pollDesc).wait(0x9746a6f4, 0x72, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:84 +0x8c fp=0x97264e7c sp=0x97264e68
internal/poll.(*pollDesc).waitRead(0x9746a6f4, 0xffffffff, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_poll_runtime.go:89 +0x29 fp=0x97264e90 sp=0x97264e7c
internal/poll.(*FD).Accept(0x9746a6e0, 0x0, 0x0, 0x0, 0x85de888, 0x7, 0x8866400, 0x97252718)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/internal/poll/fd_unix.go:349 +0x181 fp=0x97264eb8 sp=0x97264e90
net.(*netFD).accept(0x9746a6e0, 0x0, 0x829413d, 0x805b1ec)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/fd_unix.go:243 +0x27 fp=0x97264f20 sp=0x97264eb8
net.(*UnixListener).accept(0x970d0f60, 0x81369b6, 0x85ae1c0, 0x97254760)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/unixsock_posix.go:162 +0x27 fp=0x97264f3c sp=0x97264f20
net.(*UnixListener).Accept(0x970d0f60, 0x97254740, 0x856d700, 0x885b9f8, 0x85b9160)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/unixsock.go:237 +0x3c fp=0x97264f5c sp=0x97264f3c
net/http.(*Server).Serve(0x96fb6580, 0x88683e0, 0x970d0f60, 0x0, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/net/http/server.go:2647 +0x1a1 fp=0x97264fb4 sp=0x97264f5c
github.com/lxc/lxd/lxd.(*Daemon).Init.func5.3(0x96fba150, 0x807aaee)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:1025 +0x48 fp=0x97264fd0 sp=0x97264fb4
gopkg.in/tomb%2ev2.(*Tomb).run(0x9736203c, 0x97272010)
	/lxd/build/tmp.qYqXiG8rio/go/src/gopkg.in/tomb.v2/tomb.go:163 +0x21 fp=0x97264fe8 sp=0x97264fd0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x97264fec sp=0x97264fe8
created by gopkg.in/tomb%2ev2.(*Tomb).Go
	/lxd/build/tmp.qYqXiG8rio/go/src/gopkg.in/tomb.v2/tomb.go:159 +0xae

goroutine 1235 [GC worker (idle)]:
runtime.gopark(0x8611bc8, 0x9728b980, 0x85e4627, 0x10, 0x14, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971707a0 sp=0x9717078c
runtime.gcBgMarkWorker(0x96fd2a80)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1492 +0xf9 fp=0x971707e8 sp=0x971707a0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971707ec sp=0x971707e8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 1200 [GC worker (idle)]:
runtime.gopark(0x8611bc8, 0x972ae038, 0x85e4627, 0x10, 0x14, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971a97a0 sp=0x971a978c
runtime.gcBgMarkWorker(0x96fce000)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1492 +0xf9 fp=0x971a97e8 sp=0x971a97a0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971a97ec sp=0x971a97e8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 2486 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971a3eb4 sp=0x971a3ea0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x971a3ed0 sp=0x971a3eb4
runtime.gcParkAssist(0x971a0aa8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x971a3ef0 sp=0x971a3ed0
runtime.gcAssistAlloc(0x971a0a50)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x971a3f48 sp=0x971a3ef0
runtime.mallocgc(0x8, 0x8571ca0, 0x971a0a01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x971a3f98 sp=0x971a3f48
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x971a3fac sp=0x971a3f98
database/sql.(*Rows).close(0x97313f40, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x971a3fc8 sp=0x971a3fac
database/sql.(*Rows).awaitDone(0x97313f40, 0x8868d60, 0x97322c80)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x971a3fe0 sp=0x971a3fc8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971a3fe4 sp=0x971a3fe0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 1273 [chan receive]:
runtime.gopark(0x8611c88, 0x97149fb0, 0x85e2706, 0xc, 0x971a5717, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971a574c sp=0x971a5738
runtime.goparkunlock(0x97149fb0, 0x85e2706, 0xc, 0x97130817, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x971a5768 sp=0x971a574c
runtime.chanrecv(0x8540240, 0x97149f80, 0x971a57e0, 0x1, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:513 +0x2ee fp=0x971a57b8 sp=0x971a5768
runtime.chanrecv2(0x8540240, 0x97149f80, 0x971a57e0, 0x807aaee)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:400 +0x24 fp=0x971a57d0 sp=0x971a57b8
database/sql.(*DB).connectionOpener(0x9746e960)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:837 +0x3a fp=0x971a57e8 sp=0x971a57d0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971a57ec sp=0x971a57e8
created by database/sql.Open
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:582 +0x1c4

goroutine 1202 [syscall]:
runtime.notetsleepg(0x888e60c, 0x29e638a7, 0x1, 0x1)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/lock_futex.go:205 +0x37 fp=0x971a5f98 sp=0x971a5f80
runtime.timerproc()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/time.go:209 +0x396 fp=0x971a5ff0 sp=0x971a5f98
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971a5ff4 sp=0x971a5ff0
created by runtime.addtimerLocked
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/time.go:116 +0xe0

goroutine 1201 [GC worker (idle)]:
runtime.gopark(0x8611bc8, 0x9728b970, 0x85e4627, 0x10, 0x14, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971d37a0 sp=0x971d378c
runtime.gcBgMarkWorker(0x96fcea80)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1492 +0xf9 fp=0x971d37e8 sp=0x971d37a0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971d37ec sp=0x971d37e8
created by runtime.gcBgMarkStartWorkers
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgc.go:1413 +0x68

goroutine 1198 [chan receive]:
runtime.gopark(0x8611c88, 0x970d8330, 0x85e2706, 0xc, 0x808ae17, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x97170f2c sp=0x97170f18
runtime.goparkunlock(0x970d8330, 0x85e2706, 0xc, 0x885be17, 0x3)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x97170f48 sp=0x97170f2c
runtime.chanrecv(0x853fa80, 0x970d8300, 0x0, 0x1, 0x17)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:513 +0x2ee fp=0x97170f98 sp=0x97170f48
runtime.chanrecv1(0x853fa80, 0x970d8300, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/chan.go:395 +0x24 fp=0x97170fb0 sp=0x97170f98
github.com/lxc/lxd/lxd.(*Daemon).Init.func2(0x9705ab40)
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:839 +0xd1 fp=0x97170fe8 sp=0x97170fb0
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x97170fec sp=0x97170fe8
created by github.com/lxc/lxd/lxd.(*Daemon).Init
	/lxd/build/tmp.qYqXiG8rio/go/src/github.com/lxc/lxd/lxd/daemon.go:828 +0xd34

goroutine 2488 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971ce6b4 sp=0x971ce6a0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x971ce6d0 sp=0x971ce6b4
runtime.gcParkAssist(0x97010c88)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x971ce6f0 sp=0x971ce6d0
runtime.gcAssistAlloc(0x97010c30)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x971ce748 sp=0x971ce6f0
runtime.mallocgc(0x8, 0x8571ca0, 0x97010c01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x971ce798 sp=0x971ce748
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x971ce7ac sp=0x971ce798
database/sql.(*Rows).close(0x97313f90, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x971ce7c8 sp=0x971ce7ac
database/sql.(*Rows).awaitDone(0x97313f90, 0x8868d60, 0x97322d60)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x971ce7e0 sp=0x971ce7c8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971ce7e4 sp=0x971ce7e0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 2490 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971d06b4 sp=0x971d06a0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x971d06d0 sp=0x971d06b4
runtime.gcParkAssist(0x97010f58)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x971d06f0 sp=0x971d06d0
runtime.gcAssistAlloc(0x97010f00)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x971d0748 sp=0x971d06f0
runtime.mallocgc(0x8, 0x8571ca0, 0x97010f01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x971d0798 sp=0x971d0748
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x971d07ac sp=0x971d0798
database/sql.(*Rows).close(0x973274a0, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x971d07c8 sp=0x971d07ac
database/sql.(*Rows).awaitDone(0x973274a0, 0x8868d60, 0x9732ac60)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x971d07e0 sp=0x971d07c8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971d07e4 sp=0x971d07e0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 2482 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x975516b4 sp=0x975516a0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x975516d0 sp=0x975516b4
runtime.gcParkAssist(0x970118b8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x975516f0 sp=0x975516d0
runtime.gcAssistAlloc(0x97011860)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x97551748 sp=0x975516f0
runtime.mallocgc(0x8, 0x8571ca0, 0x97011801, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x97551798 sp=0x97551748
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x975517ac sp=0x97551798
database/sql.(*Rows).close(0x97313ea0, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x975517c8 sp=0x975517ac
database/sql.(*Rows).awaitDone(0x97313ea0, 0x8868d60, 0x97322a80)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x975517e0 sp=0x975517c8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x975517e4 sp=0x975517e0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 2480 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x97551eb4 sp=0x97551ea0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x97551ed0 sp=0x97551eb4
runtime.gcParkAssist(0x970119a8)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x97551ef0 sp=0x97551ed0
runtime.gcAssistAlloc(0x97011950)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x97551f48 sp=0x97551ef0
runtime.mallocgc(0x8, 0x8571ca0, 0x97011901, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x97551f98 sp=0x97551f48
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x97551fac sp=0x97551f98
database/sql.(*Rows).close(0x97313e50, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x97551fc8 sp=0x97551fac
database/sql.(*Rows).awaitDone(0x97313e50, 0x8868d60, 0x973229e0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x97551fe0 sp=0x97551fc8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x97551fe4 sp=0x97551fe0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79

goroutine 2484 [runnable]:
runtime.gopark(0x8611c88, 0x888eddc, 0x85e3587, 0xe, 0x8097f2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:271 +0x109 fp=0x971a4eb4 sp=0x971a4ea0
runtime.goparkunlock(0x888eddc, 0x85e3587, 0xe, 0x2a, 0x2)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/proc.go:277 +0x4c fp=0x971a4ed0 sp=0x971a4eb4
runtime.gcParkAssist(0x97011b88)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:649 +0x129 fp=0x971a4ef0 sp=0x971a4ed0
runtime.gcAssistAlloc(0x97011b30)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/mgcmark.go:499 +0x1c8 fp=0x971a4f48 sp=0x971a4ef0
runtime.mallocgc(0x8, 0x8571ca0, 0x97011b01, 0x0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:579 +0x732 fp=0x971a4f98 sp=0x971a4f48
runtime.newobject(0x8571ca0, 0x813602a)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/malloc.go:803 +0x2c fp=0x971a4fac sp=0x971a4f98
database/sql.(*Rows).close(0x97313ef0, 0x8864cc0, 0x96fba150, 0x807aaee, 0x8611be0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2421 +0x27 fp=0x971a4fc8 sp=0x971a4fac
database/sql.(*Rows).awaitDone(0x97313ef0, 0x8868d60, 0x97322ba0)
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2122 +0x6f fp=0x971a4fe0 sp=0x971a4fc8
runtime.goexit()
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/runtime/asm_386.s:1615 +0x1 fp=0x971a4fe4 sp=0x971a4fe0
created by database/sql.(*Rows).initContextClose
	/lxd/build/tmp.qYqXiG8rio/go/golang/src/database/sql/sql.go:2116 +0x79
FAIL	github.com/lxc/lxd/lxd	30.323s
@brauner

This comment has been minimized.

Copy link

commented Feb 14, 2017

@aclements, the above paste is with GODEBUG=gccheckmark=1 set and on a i386 runner. Her is another link to a paste with the same GODEBUG=gcheckmark=1 set but on a amd64 runner: https://paste.ubuntu.com/23992323/.

@brauner

This comment has been minimized.

Copy link

commented Feb 14, 2017

@rsc, it does not seem to be reproducible with Go 1.7.5.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2017

@brauner, those traces are great, I see what's going on and I've reproduced at master. Checking to see what earlier versions the problem exists in.

But you were seeing other failures (without checkmark) in Go 1.7. Are they gone in Go 1.7.5? Are they gone in Go 1.8rc3?

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2017

@brauner, your bug is #19078. Let's move discussion of that problem there.

If anyone has some reliable reproduction cases for "sweep increased allocation count" in Go 1.7.5 or Go 1.8rc3, please let us know.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2017

@brauner, thanks, so just to confirm, you originally posted a failure beginning:

runtime: nelems=128 nfree=86 nalloc=42 previous allocCount=40 nfreed=65534
fatal error: sweep increased allocation count

In a followup then you said that was from Go 1.7, and in your more recent followup you said that with Go 1.7.5 you are unable to reproduce that specific failure anymore.

Can you confirm I've got that right? Thanks.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2017

Or maybe I just replied to the wrong issue...

@brauner

This comment has been minimized.

Copy link

commented Feb 14, 2017

@rsc, so far we only managed to reproduce it with current Go master (2017-02-14). All other Go versions build fine.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2017

@brauner, but you had the original report about "sweep increased allocation count". Was that Go 1.7?

@brauner

This comment has been minimized.

Copy link

commented Feb 14, 2017

@rsc, no. The original report was Go master (2017-02-14).

@aclements

This comment has been minimized.

Copy link
Member

commented Feb 14, 2017

@cynecx, are you able to reproduce the issue you saw? If so, how often? Can you run with GODEBUG=gccheckmark=1 set?

@cynecx

This comment has been minimized.

Copy link
Author

commented Feb 15, 2017

Sorry about the delay.

> GODEBUG=gccheckmark=1 go get -u -v github.com/nsf/gocode
runtime:greyobject: checkmarks finds unexpected unmarked object obj=0xc4202b2d80
runtime: found obj at *(0xc42029a480+0x400)
base=0xc42029a480 k=0x621014d s.base()=0xc42029a000 s.limit=0xc42029bf80 s.sizeclass=32 s.elemsize=1152
*(base+0) = 0xc4203f7340
*(base+8) = 0x17
*(base+16) = 0xc4203e2997
*(base+24) = 0x7
*(base+32) = 0x0
*(base+40) = 0x0
*(base+48) = 0xc4203e37a7
*(base+56) = 0x7
*(base+64) = 0xc420367720
*(base+72) = 0x50
*(base+80) = 0xc4203001e0
*(base+88) = 0x25
*(base+96) = 0x0
*(base+104) = 0x0
*(base+112) = 0x101
*(base+120) = 0x0
*(base+128) = 0x0
*(base+136) = 0x84c7df
*(base+144) = 0xb
*(base+152) = 0x0
*(base+160) = 0x0
*(base+168) = 0x0
*(base+176) = 0xc4203df980
*(base+184) = 0x7
*(base+192) = 0x8
*(base+200) = 0x0
*(base+208) = 0x0
*(base+216) = 0x0
*(base+224) = 0xc42037c2b0
*(base+232) = 0x1
*(base+240) = 0x1
*(base+248) = 0x0
*(base+256) = 0x0
*(base+264) = 0x0
*(base+272) = 0x0
*(base+280) = 0x0
*(base+288) = 0x0
*(base+296) = 0x0
*(base+304) = 0x0
*(base+312) = 0x0
*(base+320) = 0x0
*(base+328) = 0x0
*(base+336) = 0x0
*(base+344) = 0x0
*(base+352) = 0x0
*(base+360) = 0x0
*(base+368) = 0xc42037c160
*(base+376) = 0x1
*(base+384) = 0x1
*(base+392) = 0x0
*(base+400) = 0x0
*(base+408) = 0x0
*(base+416) = 0x0
*(base+424) = 0x0
*(base+432) = 0x0
*(base+440) = 0x0
*(base+448) = 0x0
*(base+456) = 0x0
*(base+464) = 0x0
*(base+472) = 0x0
*(base+480) = 0x0
*(base+488) = 0x0
*(base+496) = 0x0
*(base+504) = 0x0
*(base+512) = 0x0
*(base+520) = 0x0
*(base+528) = 0x0
*(base+536) = 0x0
*(base+544) = 0x0
*(base+552) = 0x0
*(base+560) = 0x0
*(base+568) = 0x0
*(base+576) = 0x0
*(base+584) = 0x0
*(base+592) = 0x0
*(base+600) = 0x0
*(base+608) = 0xc4203ead40
*(base+616) = 0x4
*(base+624) = 0x4
*(base+632) = 0xc4201f24d0
*(base+640) = 0xb
*(base+648) = 0xb
*(base+656) = 0x0
*(base+664) = 0x0
*(base+672) = 0x0
*(base+680) = 0x0
*(base+688) = 0x0
*(base+696) = 0xc4203e3a90
*(base+704) = 0x1
*(base+712) = 0x1
*(base+720) = 0xa4b220
*(base+728) = 0x0
*(base+736) = 0x0
*(base+744) = 0xc4203df680
*(base+752) = 0x6
*(base+760) = 0x8
*(base+768) = 0xc42015e180
*(base+776) = 0xc
*(base+784) = 0xc
*(base+792) = 0xc420455180
*(base+800) = 0x0
*(base+808) = 0x0
*(base+816) = 0xc4203ead80
*(base+824) = 0x5
*(base+832) = 0x8
*(base+840) = 0xc420382080
*(base+848) = 0xb
*(base+856) = 0x10
*(base+864) = 0xc4200942a0
*(base+872) = 0xe
*(base+880) = 0xe
*(base+888) = 0xc42037c400
*(base+896) = 0x1
*(base+904) = 0x1
*(base+912) = 0xc42010a3c0
*(base+920) = 0xf
*(base+928) = 0xf
*(base+936) = 0xc4203001e0
*(base+944) = 0x25
*(base+952) = 0x0
*(base+960) = 0xc4203d9d20
*(base+968) = 0x18
*(base+976) = 0x0
*(base+984) = 0x0
*(base+992) = 0x0
*(base+1000) = 0x0
*(base+1008) = 0x0
*(base+1016) = 0x0
*(base+1024) = 0xc4202b2d80 <==
*(base+1032) = 0x28
*(base+1040) = 0x0
*(base+1048) = 0x0
*(base+1056) = 0x0
*(base+1064) = 0x0
*(base+1072) = 0x0
*(base+1080) = 0x0
*(base+1088) = 0x0
*(base+1096) = 0x0
*(base+1104) = 0x0
*(base+1112) = 0x0
*(base+1120) = 0x0
*(base+1128) = 0x0
*(base+1136) = 0x0
*(base+1144) = 0x0
obj=0xc4202b2d80 k=0x6210159 s.base()=0xc4202b2000 s.limit=0xc4202b3fe0 s.sizeclass=4 s.elemsize=48
*(obj+0) = 0x3637663236366333
*(obj+8) = 0x3364306236613462
*(obj+16) = 0x3236316334383264
*(obj+24) = 0x6563396439333035
*(obj+32) = 0x3163643235633364
*(obj+40) = 0x0
fatal error: checkmark found unmarked object

runtime stack:
runtime.throw(0x855ea0, 0x1f)
    /usr/lib/go/src/runtime/panic.go:566 +0x95 fp=0x7f3abb7fdc50 sp=0x7f3abb7fdc30
runtime.greyobject(0xc4202b2d80, 0xc42029a480, 0x400, 0xc41ffea693, 0xc400000000, 0x7f3ac234f290, 0xc420027c28, 0x48)
    /usr/lib/go/src/runtime/mgcmark.go:1223 +0x309 fp=0x7f3abb7fdc80 sp=0x7f3abb7fdc50
runtime.scanobject(0xc42029a480, 0xc420027c28)
    /usr/lib/go/src/runtime/mgcmark.go:1176 +0x1fa fp=0x7f3abb7fdd00 sp=0x7f3abb7fdc80
runtime.gcDrain(0xc420027c28, 0x0)
    /usr/lib/go/src/runtime/mgcmark.go:999 +0x15c fp=0x7f3abb7fdd38 sp=0x7f3abb7fdd00
runtime.gchelper()
    /usr/lib/go/src/runtime/mgc.go:1827 +0x89 fp=0x7f3abb7fdd78 sp=0x7f3abb7fdd38
runtime.stopm()
    /usr/lib/go/src/runtime/proc.go:1597 +0xdf fp=0x7f3abb7fdda0 sp=0x7f3abb7fdd78
runtime.findrunnable(0xc420026a00, 0x0)
    /usr/lib/go/src/runtime/proc.go:2021 +0x228 fp=0x7f3abb7fde50 sp=0x7f3abb7fdda0
runtime.schedule()
    /usr/lib/go/src/runtime/proc.go:2120 +0x14c fp=0x7f3abb7fde90 sp=0x7f3abb7fde50
runtime.park_m(0xc420130340)
    /usr/lib/go/src/runtime/proc.go:2183 +0x123 fp=0x7f3abb7fdec0 sp=0x7f3abb7fde90
runtime.mcall(0x0)
    /usr/lib/go/src/runtime/asm_amd64.s:240 +0x5b fp=0x7f3abb7fded0 sp=0x7f3abb7fdec0

goroutine 1 [runnable]:
runtime.mallocgc(0x7, 0x0, 0xc400000000, 0x487fbd)
    /usr/lib/go/src/runtime/malloc.go:529 fp=0xc42043df68 sp=0xc42043df60
runtime.rawstring(0x7, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/runtime/string.go:289 +0x85 fp=0xc42043dfa0 sp=0xc42043df68
runtime.rawstringtmp(0x0, 0x7, 0x4, 0x8, 0x4, 0xc42023a088, 0xc42043e058)
    /usr/lib/go/src/runtime/string.go:111 +0xa1 fp=0xc42043dfe0 sp=0xc42043dfa0
runtime.slicebytetostring(0x0, 0xc4204d41a0, 0x7, 0x60, 0xc42043e070, 0x633823)
    /usr/lib/go/src/runtime/string.go:93 +0x3e fp=0xc42043e038 sp=0xc42043dfe0
go/scanner.(*Scanner).scanIdentifier(0xc4202f6020, 0x1, 0x25e6)
    /usr/lib/go/src/go/scanner/scanner.go:270 +0xc2 fp=0xc42043e080 sp=0xc42043e038
go/scanner.(*Scanner).Scan(0xc4202f6020, 0x10, 0x7f9d20, 0x1, 0xc4202f0150)
    /usr/lib/go/src/go/scanner/scanner.go:609 +0xb01 fp=0xc42043e108 sp=0xc42043e080
go/parser.(*parser).next0(0xc4202f6000)
    /usr/lib/go/src/go/parser/parser.go:257 +0x78 fp=0xc42043e1d0 sp=0xc42043e108
go/parser.(*parser).consumeComment(0xc4202f6000, 0x25b3, 0x3)
    /usr/lib/go/src/go/parser/parser.go:275 +0xe7 fp=0xc42043e208 sp=0xc42043e1d0
go/parser.(*parser).consumeCommentGroup(0xc4202f6000, 0x1, 0x0, 0xc42043e2f0)
    /usr/lib/go/src/go/parser/parser.go:290 +0xd2 fp=0xc42043e2a0 sp=0xc42043e208
go/parser.(*parser).next(0xc4202f6000)
    /usr/lib/go/src/go/parser/parser.go:340 +0x106 fp=0xc42043e300 sp=0xc42043e2a0
go/parser.(*parser).init(0xc4202f6000, 0xc42016e980, 0xc4202f0150, 0x24, 0xc4204d4100, 0xf0, 0x100, 0x6)
    /usr/lib/go/src/go/parser/parser.go:85 +0x149 fp=0xc42043e358 sp=0xc42043e300
go/parser.ParseFile(0xc42016e980, 0xc4202f0150, 0x24, 0x7bd6a0, 0xc4201402a0, 0x6, 0x0, 0x0, 0x0)
    /usr/lib/go/src/go/parser/interface.go:120 +0x1d3 fp=0xc42043e3e0 sp=0xc42043e358
go/build.(*Context).Import(0xa2ea40, 0xc42014f4c1, 0x3, 0xc4201507b0, 0x2d, 0x4, 0xc4204a6d01, 0xc42043ef50, 0x47fc61)
    /usr/lib/go/src/go/build/build.go:739 +0xcac fp=0xc42043ee30 sp=0xc42043e3e0
main.loadImport(0xc42014f4c1, 0x3, 0xc4201507b0, 0x2d, 0xc4201ae000, 0xc42014a640, 0xc4201d8660, 0x1, 0x1, 0x1, ...)
    /usr/lib/go/src/cmd/go/pkg.go:365 +0x3b9 fp=0xc42043ef60 sp=0xc42043ee30
main.(*Package).load(0xc4201ae000, 0xc42014a640, 0xc420196a80, 0x0, 0x0, 0xc)
    /usr/lib/go/src/cmd/go/pkg.go:940 +0x12b6 fp=0xc42043f750 sp=0xc42043ef60
main.loadImport(0x7fff8d43d439, 0x15, 0xc420010064, 0x13, 0x0, 0xc42014a640, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/lib/go/src/cmd/go/pkg.go:374 +0x470 fp=0xc42043f880 sp=0xc42043f750
main.loadPackage(0x7fff8d43d439, 0x15, 0xc42014a640, 0x0)
    /usr/lib/go/src/cmd/go/pkg.go:1658 +0x28f fp=0xc42043f9e0 sp=0xc42043f880
main.download.func1(0x7fff8d43d439, 0x15, 0x0, 0xc42014e380)
    /usr/lib/go/src/cmd/go/get.go:190 +0xba fp=0xc42043fa48 sp=0xc42043f9e0
main.download(0x7fff8d43d439, 0x15, 0x0, 0xc42014a640, 0x0)
    /usr/lib/go/src/cmd/go/get.go:195 +0xf3 fp=0xc42043fc70 sp=0xc42043fa48
main.runGet(0xa27900, 0xc42000c0e0, 0x1, 0x1)
    /usr/lib/go/src/cmd/go/get.go:106 +0x145 fp=0xc42043fdb8 sp=0xc42043fc70
main.main()
    /usr/lib/go/src/cmd/go/main.go:181 +0x624 fp=0xc42043ff28 sp=0xc42043fdb8
runtime.main()
    /usr/lib/go/src/runtime/proc.go:183 +0x1f4 fp=0xc42043ff80 sp=0xc42043ff28
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42043ff88 sp=0xc42043ff80

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420044fb8 sp=0xc420044fb0

goroutine 2 [force gc (idle)]:
runtime.gopark(0x893bd0, 0xa2e260, 0x84e9a8, 0xf, 0x893a14, 0x1)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420030748 sp=0xc420030718
runtime.goparkunlock(0xa2e260, 0x84e9a8, 0xf, 0xc420000114, 0x1)
    /usr/lib/go/src/runtime/proc.go:265 +0x5e fp=0xc420030788 sp=0xc420030748
runtime.forcegchelper()
    /usr/lib/go/src/runtime/proc.go:224 +0xa8 fp=0xc4200307c0 sp=0xc420030788
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200307c8 sp=0xc4200307c0
created by runtime.init.3
    /usr/lib/go/src/runtime/proc.go:213 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x893bd0, 0xa2e460, 0x84d909, 0xd, 0x4c7f14, 0x1)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420030f38 sp=0xc420030f08
runtime.goparkunlock(0xa2e460, 0x84d909, 0xd, 0x14, 0x1)
    /usr/lib/go/src/runtime/proc.go:265 +0x5e fp=0xc420030f78 sp=0xc420030f38
runtime.bgsweep(0xc420060000)
    /usr/lib/go/src/runtime/mgcsweep.go:79 +0x12e fp=0xc420030fb8 sp=0xc420030f78
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420030fc0 sp=0xc420030fb8
created by runtime.gcenable
    /usr/lib/go/src/runtime/mgc.go:195 +0x61

goroutine 4 [finalizer wait]:
runtime.gopark(0x893bd0, 0xa4b148, 0x84e31b, 0xe, 0x14, 0x1)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420031708 sp=0xc4200316d8
runtime.goparkunlock(0xa4b148, 0x84e31b, 0xe, 0x2000000014, 0x1)
    /usr/lib/go/src/runtime/proc.go:265 +0x5e fp=0xc420031748 sp=0xc420031708
runtime.runfinq()
    /usr/lib/go/src/runtime/mfinal.go:158 +0xaf fp=0xc4200317c0 sp=0xc420031748
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200317c8 sp=0xc4200317c0
created by runtime.createfing
    /usr/lib/go/src/runtime/mfinal.go:139 +0x73

goroutine 5 [syscall]:
runtime.notetsleepg(0xa4b4a0, 0xffffffffffffffff, 0x1)
    /usr/lib/go/src/runtime/lock_futex.go:205 +0x42 fp=0xc420031f50 sp=0xc420031f20
os/signal.signal_recv(0x0)
    /usr/lib/go/src/runtime/sigqueue.go:116 +0x157 fp=0xc420031f80 sp=0xc420031f50
os/signal.loop()
    /usr/lib/go/src/os/signal/signal_unix.go:22 +0x22 fp=0xc420031fc0 sp=0xc420031f80
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420031fc8 sp=0xc420031fc0
created by os/signal.init.1
    /usr/lib/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x893a40, 0xc42013e560, 0x84edde, 0x10, 0x914, 0x0)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420032748 sp=0xc420032718
runtime.gcBgMarkWorker(0xc420025500)
    /usr/lib/go/src/runtime/mgc.go:1428 +0x108 fp=0xc4200327b8 sp=0xc420032748
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200327c0 sp=0xc4200327b8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98

goroutine 8 [garbage collection]:
runtime.systemstack_switch()
    /usr/lib/go/src/runtime/asm_amd64.s:252 fp=0xc420032d28 sp=0xc420032d20
runtime.gcMarkTermination()
    /usr/lib/go/src/runtime/mgc.go:1228 +0x193 fp=0xc420032f20 sp=0xc420032d28
runtime.gcMarkDone()
    /usr/lib/go/src/runtime/mgc.go:1158 +0x21e fp=0xc420032f48 sp=0xc420032f20
runtime.gcBgMarkWorker(0xc420022a00)
    /usr/lib/go/src/runtime/mgc.go:1505 +0x2c2 fp=0xc420032fb8 sp=0xc420032f48
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420032fc0 sp=0xc420032fb8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x893a40, 0xc4203e3a30, 0x84edde, 0x10, 0x14, 0x0)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc42002c748 sp=0xc42002c718
runtime.gcBgMarkWorker(0xc420020000)
    /usr/lib/go/src/runtime/mgc.go:1428 +0x108 fp=0xc42002c7b8 sp=0xc42002c748
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42002c7c0 sp=0xc42002c7b8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x893a40, 0xc4203e3a40, 0x84edde, 0x10, 0x14, 0x0)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc42002cf48 sp=0xc42002cf18
runtime.gcBgMarkWorker(0xc420021500)
    /usr/lib/go/src/runtime/mgc.go:1428 +0x108 fp=0xc42002cfb8 sp=0xc42002cf48
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42002cfc0 sp=0xc42002cfb8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x893a40, 0xc420167b30, 0x84edde, 0x10, 0x14, 0x0)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420033748 sp=0xc420033718
runtime.gcBgMarkWorker(0xc420024000)
    /usr/lib/go/src/runtime/mgc.go:1428 +0x108 fp=0xc4200337b8 sp=0xc420033748
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200337c0 sp=0xc4200337b8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x893a40, 0xc42013e570, 0x84edde, 0x10, 0x14, 0x0)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420464748 sp=0xc420464718
runtime.gcBgMarkWorker(0xc420026a00)
    /usr/lib/go/src/runtime/mgc.go:1428 +0x108 fp=0xc4204647b8 sp=0xc420464748
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4204647c0 sp=0xc4204647b8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x893a40, 0xc42013e580, 0x84edde, 0x10, 0x14, 0x0)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420464f48 sp=0xc420464f18
runtime.gcBgMarkWorker(0xc420028000)
    /usr/lib/go/src/runtime/mgc.go:1428 +0x108 fp=0xc420464fb8 sp=0xc420464f48
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420464fc0 sp=0xc420464fb8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x893a40, 0xc42013e590, 0x84edde, 0x10, 0x14, 0x0)
    /usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420465748 sp=0xc420465718
runtime.gcBgMarkWorker(0xc420029500)
    /usr/lib/go/src/runtime/mgc.go:1428 +0x108 fp=0xc4204657b8 sp=0xc420465748
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4204657c0 sp=0xc4204657b8
created by runtime.gcBgMarkStartWorkers
    /usr/lib/go/src/runtime/mgc.go:1349 +0x98
@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 15, 2017

Thanks. @aclements, that block is a Package struct from cmd/go, and the field at offset 1024 with the unmarked object is 'buildID string', That's only updated in one place in the code:

p.buildID = fmt.Sprintf("%x", h.Sum(nil))

which compiles to:

pkg.go:1593    0x4f32a    e841fb0900            CALL fmt.Sprintf(SB)
pkg.go:1593    0x4f32f    488b442430            MOVQ 0x30(SP), AX
pkg.go:1593    0x4f334    488b4c2428            MOVQ 0x28(SP), CX
pkg.go:1593    0x4f339    488b942408020000      MOVQ 0x208(SP), DX
pkg.go:1593    0x4f341    48898208040000        MOVQ AX, 0x408(DX)
pkg.go:1593    0x4f348    8b0572dc5f00          MOVL 0x5fdc72(IP), AX // runtime.writeBarrier
pkg.go:1593    0x4f34e    488d9a00040000        LEAQ 0x400(DX), BX
pkg.go:1593    0x4f355    84c0                  TESTL AL, AL
pkg.go:1593    0x4f357    7517                  JNE 0x4f370
pkg.go:1593    0x4f359    48898a00040000        MOVQ CX, 0x400(DX)
pkg.go:1594    0x4f360    488bac24f8010000      MOVQ 0x1f8(SP), BP
pkg.go:1594    0x4f368    4881c400020000        ADDQ $0x200, SP
pkg.go:1594    0x4f36f    c3                    RET
pkg.go:1593    0x4f370    48891c24              MOVQ BX, 0(SP)
pkg.go:1593    0x4f374    48894c2408            MOVQ CX, 0x8(SP)
pkg.go:1593    0x4f379    e8929b0300            CALL runtime.writebarrierptr(SB)
pkg.go:1550    0x4f37e    488b942408020000      MOVQ 0x208(SP), DX
pkg.go:1594    0x4f386    ebd8                  JMP 0x4f360

This looks right to me, provided runtime.writeBarrier and runtime.writerbarrierptr are correct.

There is another write for binary-only packages but there aren't any binary-only packages involved here and p.BinaryOnly (the bool at offset 168) is false.

@greyltc

This comment has been minimized.

Copy link

commented Feb 26, 2017

I've got this too. Very repeatable. Let me know if there's anything I can do to help.
go version = go version go1.8rc3 linux/amd64

$ rclone mount ndCrypt:mount ~/remote
runtime: nelems=128 nfree=88 nalloc=40 previous allocCount=39 nfreed=65535
fatal error: sweep increased allocation count

runtime stack:
runtime.throw(0xc291ac, 0x20)
	/usr/lib/go/src/runtime/panic.go:566 +0x95
runtime.(*mspan).sweep(0x7f01b212bda0, 0xd00000000, 0xc400000101)
	/usr/lib/go/src/runtime/mgcsweep.go:287 +0x7ab
runtime.sweepone(0x432796)
	/usr/lib/go/src/runtime/mgcsweep.go:112 +0xfe
runtime.gosweepone.func1()
	/usr/lib/go/src/runtime/mgcsweep.go:124 +0x2b
runtime.systemstack(0xc42001a000)
	/usr/lib/go/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
	/usr/lib/go/src/runtime/proc.go:1079

goroutine 3 [running]:
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_amd64.s:252 fp=0xc42002af48 sp=0xc42002af40
runtime.gosweepone(0x0)
	/usr/lib/go/src/runtime/mgcsweep.go:125 +0x4a fp=0xc42002af78 sp=0xc42002af48
runtime.bgsweep(0xc420058000)
	/usr/lib/go/src/runtime/mgcsweep.go:66 +0xbb fp=0xc42002afb8 sp=0xc42002af78
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42002afc0 sp=0xc42002afb8
created by runtime.gcenable
	/usr/lib/go/src/runtime/mgc.go:195 +0x61

goroutine 1 [chan receive]:
github.com/ncw/rclone/cmd/mount.Mount(0xfdb800, 0xc4213f4ff0, 0x7ffe35c77765, 0x11, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/mount/mount.go:163 +0x143
github.com/ncw/rclone/cmd/mount.glob..func1(0x1004fc0, 0xc4201bf880, 0x2, 0x2)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/mount/mount.go:129 +0xbf
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).execute(0x1004fc0, 0xc4201bf6e0, 0x2, 0x2, 0x1004fc0, 0xc4201bf6e0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.go:636 +0x443
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1002760, 0x20, 0xc4201bf620, 0xc42016b400)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.go:722 +0x367
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).Execute(0x1002760, 0x0, 0x40)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.go:681 +0x2b
main.main()
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/rclone.go:15 +0x31

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
	/usr/lib/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
	/usr/lib/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/lib/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 9 [IO wait]:
net.runtime_pollWait(0x7f01b20b0138, 0x72, 0x3)
	/usr/lib/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420256220, 0x72, 0xc42003f5e8, 0xc420012220)
	/usr/lib/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420256220, 0xfd2fe0, 0xc420012220)
	/usr/lib/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4202561c0, 0xc42035e000, 0x1000, 0x1000, 0x0, 0xfd2fe0, 0xc420012220)
	/usr/lib/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4201680a8, 0xc42035e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/net.go:173 +0x70
net.(Conn).Read-fm(0xc42035e000, 0x1000, 0x1000, 0xc4213fa1cc, 0x4, 0xa)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/http.go:68 +0x4d
github.com/ncw/rclone/fs.(*timeoutConn).readOrWrite(0xc420164000, 0xc42003f770, 0xc42035e000, 0x1000, 0x1000, 0x0, 0x5, 0x6)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/http.go:58 +0x69
github.com/ncw/rclone/fs.(*timeoutConn).Read(0xc420164000, 0xc42035e000, 0x1000, 0x1000, 0xbb0dc0, 0x0, 0x7f01b20ea800)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/http.go:68 +0x8b
crypto/tls.(*block).readFromUntil(0xc420196510, 0x7f01b20ea800, 0xc420164000, 0x5, 0xc420164000, 0x28)
	/usr/lib/go/src/crypto/tls/conn.go:476 +0x91
crypto/tls.(*Conn).readRecord(0xc4201d6000, 0xc7bc17, 0xc4201d6108, 0xc4212af860)
	/usr/lib/go/src/crypto/tls/conn.go:578 +0xc4
crypto/tls.(*Conn).Read(0xc4201d6000, 0xc42025a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/crypto/tls/conn.go:1113 +0x116
net/http.(*persistConn).Read(0xc42000ba00, 0xc42025a000, 0x1000, 0x1000, 0x6c1420, 0xc42003fb58, 0x4045ed)
	/usr/lib/go/src/net/http/transport.go:1261 +0x154
bufio.(*Reader).fill(0xc420019140)
	/usr/lib/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc420019140, 0x1, 0xc42003fbbd, 0x1, 0x0, 0xc4213b6960, 0x0)
	/usr/lib/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc42000ba00)
	/usr/lib/go/src/net/http/transport.go:1418 +0x1a1
created by net/http.(*Transport).dialConn
	/usr/lib/go/src/net/http/transport.go:1062 +0x4e9

goroutine 10 [select]:
net/http.(*persistConn).writeLoop(0xc42000ba00)
	/usr/lib/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
	/usr/lib/go/src/net/http/transport.go:1063 +0x50e

goroutine 224 [select]:
net/http.(*persistConn).readLoop(0xc4212b8500)
	/usr/lib/go/src/net/http/transport.go:1541 +0x9c9
created by net/http.(*Transport).dialConn
	/usr/lib/go/src/net/http/transport.go:1062 +0x4e9

goroutine 66 [syscall]:
syscall.Syscall(0x0, 0x8, 0xc420634000, 0x21000, 0xc421340000, 0xc42011c890, 0x4111ed)
	/usr/lib/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x8, 0xc420634000, 0x21000, 0x21000, 0xc4213b6648, 0xc400000001, 0xc400000001)
	/usr/lib/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55
syscall.Read(0x8, 0xc420634000, 0x21000, 0x21000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/syscall/syscall_unix.go:161 +0x49
github.com/ncw/rclone/vendor/bazil.org/fuse.(*Conn).ReadRequest(0xc4213b6600, 0xc7a8d8, 0xc421332000, 0xfd6de0, 0xc420256000)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/bazil.org/fuse/fuse.go:552 +0xcc
github.com/ncw/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve(0xc421332000, 0xfcf420, 0xc4213fa120, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/bazil.org/fuse/fs/serve.go:414 +0x364
github.com/ncw/rclone/vendor/bazil.org/fuse/fs.Serve(0xc4213b6600, 0xfcf420, 0xc4213fa120, 0xc4200121c0, 0x432838)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/bazil.org/fuse/fs/serve.go:435 +0x171
github.com/ncw/rclone/cmd/mount.mount.func1(0xc4213b6600, 0xc4213fa120, 0xc421350000)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/mount/fs.go:91 +0x45
created by github.com/ncw/rclone/cmd/mount.mount
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/mount/fs.go:97 +0x298

goroutine 306 [runnable]:
github.com/ncw/rclone/crypt.(*nonce).increment(0xc4203d60c8)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/crypt/cipher.go:385
github.com/ncw/rclone/crypt.(*decrypter).fillBuffer(0xc4203d60b0, 0xc7bcc0, 0xc4203d60b0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/crypt/cipher.go:591 +0x206
github.com/ncw/rclone/crypt.(*decrypter).Read(0xc4203d60b0, 0xc420686000, 0x10000, 0x10000, 0x0, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/crypt/cipher.go:604 +0x16a
github.com/ncw/rclone/fs.(*Account).read(0xc420149180, 0x7f01b20eb388, 0xc4203d60b0, 0xc420686000, 0x10000, 0x10000, 0x10000, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/accounting.go:384 +0xa8
github.com/ncw/rclone/fs.(*Account).Read(0xc420149180, 0xc420686000, 0x10000, 0x10000, 0x0, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/accounting.go:414 +0x15e
io.ReadAtLeast(0xfcf520, 0xc420149180, 0xc420676000, 0x20000, 0x20000, 0x20000, 0xad5160, 0xc4213ec028, 0xc420256040)
	/usr/lib/go/src/io/io.go:307 +0xa4
io.ReadFull(0xfcf520, 0xc420149180, 0xc420676000, 0x20000, 0x20000, 0x20000, 0x2, 0x7f01b2198b10)
	/usr/lib/go/src/io/io.go:325 +0x58
github.com/ncw/rclone/cmd/mount.(*ReadFileHandle).Read(0xc4201947c0, 0x7f01b20b0e40, 0xc42000c5c0, 0xc420256000, 0xc4213f8080, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/mount/read.go:112 +0x298
github.com/ncw/rclone/vendor/bazil.org/fuse/fs.(*Server).handleRequest(0xc421332000, 0x7f01b20b0e40, 0xc42000c5c0, 0xfcf460, 0xc420288a80, 0xc420288bc0, 0xfd6de0, 0xc420256000, 0xc420119eb8, 0x0, ...)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/bazil.org/fuse/fs/serve.go:1249 +0x98c
github.com/ncw/rclone/vendor/bazil.org/fuse/fs.(*Server).serve(0xc421332000, 0xfd6de0, 0xc420256000)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/bazil.org/fuse/fs/serve.go:878 +0x4a3
github.com/ncw/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve.func1(0xc421332000, 0xfd6de0, 0xc420256000)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/bazil.org/fuse/fs/serve.go:425 +0x6e
created by github.com/ncw/rclone/vendor/bazil.org/fuse/fs.(*Server).Serve
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/bazil.org/fuse/fs/serve.go:426 +0x3e4

goroutine 225 [select]:
net/http.(*persistConn).writeLoop(0xc4212b8500)
	/usr/lib/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
	/usr/lib/go/src/net/http/transport.go:1063 +0x50e

goroutine 227 [select]:
github.com/ncw/rclone/fs.(*Account).averageLoop(0xc420149180)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/accounting.go:358 +0x2da
created by github.com/ncw/rclone/fs.NewAccountSizeName
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/accounting.go:290 +0x1ac

and with GODEBUG=gccheckmark=1:

$ GODEBUG=gccheckmark=1 rclone mount ndCrypt:mount ~/remote
runtime:greyobject: checkmarks finds unexpected unmarked object obj=0xc421305bc0
runtime: found obj at *(0xc4201e5680+0xb8)
base=0xc4201e5680 k=0x62100f2 s.base()=0xc4201e4000 s.limit=0xc4201e5f80 s.sizeclass=32 s.elemsize=1152
 *(base+0) = 0xc420150700
 *(base+8) = 0x358
 *(base+16) = 0x35a
 *(base+24) = 0xc420150704
 *(base+32) = 0x240
 *(base+40) = 0x356
 *(base+48) = 0xc4201507c9
 *(base+56) = 0x126
 *(base+64) = 0x291
 *(base+72) = 0xc420150785
 *(base+80) = 0x44
 *(base+88) = 0x2d5
 *(base+96) = 0xc420150721
 *(base+104) = 0x44
 *(base+112) = 0x339
 *(base+120) = 0xc420150958
 *(base+128) = 0x100
 *(base+136) = 0x102
 *(base+144) = 0x3
 *(base+152) = 0x1
 *(base+160) = 0xaac9a0
 *(base+168) = 0xc42011ec30
 *(base+176) = 0x3
 *(base+184) = 0xc421305bc0 <==
 *(base+192) = 0xc42011ee40
 *(base+200) = 0x1
 *(base+208) = 0x1
 *(base+216) = 0xc42011ee50
 *(base+224) = 0x1
 *(base+232) = 0x1
 *(base+240) = 0x0
 *(base+248) = 0x0
 *(base+256) = 0x0
 *(base+264) = 0x0
 *(base+272) = 0x0
 *(base+280) = 0x0
 *(base+288) = 0x0
 *(base+296) = 0x0
 *(base+304) = 0x0
 *(base+312) = 0x0
 *(base+320) = 0x0
 *(base+328) = 0x0
 *(base+336) = 0x0
 *(base+344) = 0x0
 *(base+352) = 0x0
 *(base+360) = 0x0
 *(base+368) = 0x0
 *(base+376) = 0xc42130a4a0
 *(base+384) = 0x12
 *(base+392) = 0xc420239400
 *(base+400) = 0x3
 *(base+408) = 0x4
 *(base+416) = 0x0
 *(base+424) = 0x0
 *(base+432) = 0x0
 *(base+440) = 0xc42011ee60
 *(base+448) = 0x1
 *(base+456) = 0x1
 *(base+464) = 0xc42011ee70
 *(base+472) = 0x1
 *(base+480) = 0x1
 *(base+488) = 0x0
 *(base+496) = 0x0
 *(base+504) = 0x0
 *(base+512) = 0x0
 *(base+520) = 0x0
 *(base+528) = 0x0
 *(base+536) = 0x0
 *(base+544) = 0x0
 *(base+552) = 0x0
 *(base+560) = 0x0
 *(base+568) = 0x0
 *(base+576) = 0x0
 *(base+584) = 0x0
 *(base+592) = 0x0
 *(base+600) = 0x0
 *(base+608) = 0x0
 *(base+616) = 0x0
 *(base+624) = 0xc42130a280
 *(base+632) = 0x12
 *(base+640) = 0xc4202394a0
 *(base+648) = 0x3
 *(base+656) = 0x4
 *(base+664) = 0x0
 *(base+672) = 0x0
 *(base+680) = 0x0
 *(base+688) = 0xeb47bbd40
 *(base+696) = 0x0
 *(base+704) = 0x1000940
 *(base+712) = 0xeda1a5ac0
 *(base+720) = 0x0
 *(base+728) = 0x1000940
 *(base+736) = 0x0
 *(base+744) = 0xc42019eb60
 *(base+752) = 0x3
 *(base+760) = 0x4
 *(base+768) = 0x0
 *(base+776) = 0x0
 *(base+784) = 0x0
 *(base+792) = 0x0
 *(base+800) = 0x0
 *(base+808) = 0x0
 *(base+816) = 0x0
 *(base+824) = 0x0
 *(base+832) = 0x0
 *(base+840) = 0x0
 *(base+848) = 0x0
 *(base+856) = 0x0
 *(base+864) = 0x101
 *(base+872) = 0xffffffffffffffff
 *(base+880) = 0x0
 *(base+888) = 0xc42130a520
 *(base+896) = 0x14
 *(base+904) = 0x14
 *(base+912) = 0xc42130a5c0
 *(base+920) = 0x14
 *(base+928) = 0x14
 *(base+936) = 0x0
 *(base+944) = 0x0
 *(base+952) = 0x0
 *(base+960) = 0x0
 *(base+968) = 0x0
 *(base+976) = 0x0
 *(base+984) = 0x0
 *(base+992) = 0x0
 *(base+1000) = 0x0
 *(base+1008) = 0x0
 *(base+1016) = 0x0
 ...
obj=0xc421305bc0 k=0x6210982 s.base()=0xc421304000 s.limit=0xc421306000 s.sizeclass=3 s.elemsize=32
 *(obj+0) = 0x0
 *(obj+8) = 0xc42020fd10
 *(obj+16) = 0x1
 *(obj+24) = 0x5
fatal error: checkmark found unmarked object

runtime stack:
runtime.throw(0xc280c2, 0x1f)
	/usr/lib/go/src/runtime/panic.go:566 +0x95 fp=0x7f7fbe8f3c50 sp=0x7f7fbe8f3c30
runtime.greyobject(0xc421305bc0, 0xc4201e5680, 0xb8, 0xc41ff67d21, 0xc400000000, 0x7f7fc000ec00, 0xc420021c28, 0xde)
	/usr/lib/go/src/runtime/mgcmark.go:1217 +0x309 fp=0x7f7fbe8f3c80 sp=0x7f7fbe8f3c50
runtime.scanobject(0xc4201e5680, 0xc420021c28)
	/usr/lib/go/src/runtime/mgcmark.go:1170 +0x1fa fp=0x7f7fbe8f3d00 sp=0x7f7fbe8f3c80
runtime.gcDrain(0xc420021c28, 0x0)
	/usr/lib/go/src/runtime/mgcmark.go:993 +0x15c fp=0x7f7fbe8f3d38 sp=0x7f7fbe8f3d00
runtime.gchelper()
	/usr/lib/go/src/runtime/mgc.go:1819 +0x89 fp=0x7f7fbe8f3d78 sp=0x7f7fbe8f3d38
runtime.stopm()
	/usr/lib/go/src/runtime/proc.go:1597 +0xdf fp=0x7f7fbe8f3da0 sp=0x7f7fbe8f3d78
runtime.findrunnable(0xc420023500, 0x0)
	/usr/lib/go/src/runtime/proc.go:2021 +0x228 fp=0x7f7fbe8f3e50 sp=0x7f7fbe8f3da0
runtime.schedule()
	/usr/lib/go/src/runtime/proc.go:2120 +0x14c fp=0x7f7fbe8f3e90 sp=0x7f7fbe8f3e50
runtime.park_m(0xc42014f860)
	/usr/lib/go/src/runtime/proc.go:2183 +0x123 fp=0x7f7fbe8f3ec0 sp=0x7f7fbe8f3e90
runtime.mcall(0x0)
	/usr/lib/go/src/runtime/asm_amd64.s:240 +0x5b fp=0x7f7fbe8f3ed0 sp=0x7f7fbe8f3ec0

goroutine 1 [runnable]:
reflect.Value.Cap(0xac3bc0, 0xc4203a6240, 0x197, 0x197)
	/usr/lib/go/src/reflect/value.go:663 fp=0xc420398c38 sp=0xc420398c30
encoding/json.(*decodeState).array(0xc42128a6a8, 0xac3bc0, 0xc4203a6240, 0x197)
	/usr/lib/go/src/encoding/json/decode.go:503 +0x58f fp=0xc420398d58 sp=0xc420398c38
encoding/json.(*decodeState).value(0xc42128a6a8, 0xac3bc0, 0xc4203a6240, 0x197)
	/usr/lib/go/src/encoding/json/decode.go:365 +0x25f fp=0xc420398dd8 sp=0xc420398d58
encoding/json.(*decodeState).object(0xc42128a6a8, 0xb185e0, 0xc4203a00b0, 0x196)
	/usr/lib/go/src/encoding/json/decode.go:696 +0x1080 fp=0xc420399068 sp=0xc420398dd8
encoding/json.(*decodeState).value(0xc42128a6a8, 0xb185e0, 0xc4203a00b0, 0x196)
	/usr/lib/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc4203990e8 sp=0xc420399068
encoding/json.(*decodeState).array(0xc42128a6a8, 0xac3b80, 0xc42035e000, 0x197)
	/usr/lib/go/src/encoding/json/decode.go:519 +0x1e1 fp=0xc420399208 sp=0xc4203990e8
encoding/json.(*decodeState).value(0xc42128a6a8, 0xac3b80, 0xc42035e000, 0x197)
	/usr/lib/go/src/encoding/json/decode.go:365 +0x25f fp=0xc420399288 sp=0xc420399208
encoding/json.(*decodeState).object(0xc42128a6a8, 0xaa6320, 0xc420086020, 0x16)
	/usr/lib/go/src/encoding/json/decode.go:696 +0x1080 fp=0xc420399518 sp=0xc420399288
encoding/json.(*decodeState).value(0xc42128a6a8, 0xaa6320, 0xc420086020, 0x16)
	/usr/lib/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc420399598 sp=0xc420399518
encoding/json.(*decodeState).unmarshal(0xc42128a6a8, 0xaa6320, 0xc420086020, 0x0, 0x0)
	/usr/lib/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc420399610 sp=0xc420399598
encoding/json.(*Decoder).Decode(0xc42128a680, 0xaa6320, 0xc420086020, 0x7f7fbc694000, 0xc4203580e0)
	/usr/lib/go/src/encoding/json/stream.go:67 +0x15a fp=0xc420399658 sp=0xc420399610
github.com/ncw/rclone/vendor/google.golang.org/api/drive/v2.(*AboutGetCall).Do(0xc420399730, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/google.golang.org/api/drive/v2/drive-gen.go:2633 +0x388 fp=0xc4203996d0 sp=0xc420399658
github.com/ncw/rclone/drive.NewFs.func1(0xc4201882a0, 0xc420188358, 0xc41fff8192)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/drive/drive.go:308 +0xe1 fp=0xc420399778 sp=0xc4203996d0
github.com/ncw/rclone/pacer.(*Pacer).call(0xc4201882a0, 0xc4200fcda0, 0xa, 0xb6eaa0, 0xc42017a101)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/pacer/pacer.go:334 +0x89 fp=0xc420399830 sp=0xc420399778
github.com/ncw/rclone/pacer.(*Pacer).Call(0xc4201882a0, 0xc4200fcda0, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/pacer/pacer.go:357 +0x66 fp=0xc420399870 sp=0xc420399830
github.com/ncw/rclone/drive.NewFs(0xc4201a23f0, 0x7, 0xc4201a23f8, 0x21, 0x7, 0xc4201a23f8, 0x21, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/drive/drive.go:310 +0x464 fp=0xc420399960 sp=0xc420399870
github.com/ncw/rclone/fs.NewFs(0xc4201a23f0, 0x29, 0x2, 0xc4201a23f0, 0x29, 0xc4212c3a18)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/fs.go:459 +0x7d fp=0xc4203999b8 sp=0xc420399960
github.com/ncw/rclone/crypt.NewFs(0x7ffdb718d741, 0x7, 0x7ffdb718d749, 0x5, 0x7, 0x7ffdb718d749, 0x5, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/crypt/crypt.go:80 +0x588 fp=0xc420399b30 sp=0xc4203999b8
github.com/ncw/rclone/fs.NewFs(0x7ffdb718d741, 0xd, 0xc42021bc50, 0x461a03, 0xc18e39, 0xe)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/fs.go:459 +0x7d fp=0xc420399b88 sp=0xc420399b30
github.com/ncw/rclone/cmd.newFsDst(0x7ffdb718d741, 0xd, 0x0, 0x0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/cmd.go:148 +0x4d fp=0xc420399c18 sp=0xc420399b88
github.com/ncw/rclone/cmd.NewFsDst(0xc4201cd520, 0x2, 0x2, 0xc4201cd520, 0x2)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/cmd.go:222 +0x49 fp=0xc420399c68 sp=0xc420399c18
github.com/ncw/rclone/cmd/mount.glob..func1(0x1004fc0, 0xc4201cd520, 0x2, 0x2)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/cmd/mount/mount.go:128 +0x81 fp=0xc420399cb8 sp=0xc420399c68
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).execute(0x1004fc0, 0xc4201cd380, 0x2, 0x2, 0x1004fc0, 0xc4201cd380)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.go:636 +0x443 fp=0xc420399d70 sp=0xc420399cb8
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1002760, 0x20, 0xc4201cd2c0, 0xc420183810)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.go:722 +0x367 fp=0xc420399ec0 sp=0xc420399d70
github.com/ncw/rclone/vendor/github.com/spf13/cobra.(*Command).Execute(0x1002760, 0x0, 0x40)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/vendor/github.com/spf13/cobra/command.go:681 +0x2b fp=0xc420399ef0 sp=0xc420399ec0
main.main()
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/rclone.go:15 +0x31 fp=0xc420399f38 sp=0xc420399ef0
runtime.main()
	/usr/lib/go/src/runtime/proc.go:183 +0x1f4 fp=0xc420399f90 sp=0xc420399f38
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420399f98 sp=0xc420399f90

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42003cfb8 sp=0xc42003cfb0

goroutine 2 [force gc (idle)]:
runtime.gopark(0xc7bbb8, 0x100ccb0, 0xc19d55, 0xf, 0xc7ba14, 0x1)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc42002a748 sp=0xc42002a718
runtime.goparkunlock(0x100ccb0, 0xc19d55, 0xf, 0xc420000114, 0x1)
	/usr/lib/go/src/runtime/proc.go:265 +0x5e fp=0xc42002a788 sp=0xc42002a748
runtime.forcegchelper()
	/usr/lib/go/src/runtime/proc.go:224 +0xa8 fp=0xc42002a7c0 sp=0xc42002a788
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42002a7c8 sp=0xc42002a7c0
created by runtime.init.3
	/usr/lib/go/src/runtime/proc.go:213 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0xc7bbb8, 0x100cf20, 0xc17dd3, 0xd, 0x452514, 0x1)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc42002af38 sp=0xc42002af08
runtime.goparkunlock(0x100cf20, 0xc17dd3, 0xd, 0x14, 0x1)
	/usr/lib/go/src/runtime/proc.go:265 +0x5e fp=0xc42002af78 sp=0xc42002af38
runtime.bgsweep(0xc420058000)
	/usr/lib/go/src/runtime/mgcsweep.go:79 +0x12e fp=0xc42002afb8 sp=0xc42002af78
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42002afc0 sp=0xc42002afb8
created by runtime.gcenable
	/usr/lib/go/src/runtime/mgc.go:195 +0x61

goroutine 18 [finalizer wait]:
runtime.gopark(0xc7bbb8, 0x102a6e8, 0xc19015, 0xe, 0x14, 0x1)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420026708 sp=0xc4200266d8
runtime.goparkunlock(0x102a6e8, 0xc19015, 0xe, 0x14, 0x1)
	/usr/lib/go/src/runtime/proc.go:265 +0x5e fp=0xc420026748 sp=0xc420026708
runtime.runfinq()
	/usr/lib/go/src/runtime/mfinal.go:158 +0xaf fp=0xc4200267c0 sp=0xc420026748
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200267c8 sp=0xc4200267c0
created by runtime.createfing
	/usr/lib/go/src/runtime/mfinal.go:139 +0x73

goroutine 19 [syscall]:
runtime.notetsleepg(0x102ab20, 0xffffffffffffffff, 0x1)
	/usr/lib/go/src/runtime/lock_futex.go:205 +0x42 fp=0xc420026f50 sp=0xc420026f20
os/signal.signal_recv(0x0)
	/usr/lib/go/src/runtime/sigqueue.go:116 +0x157 fp=0xc420026f80 sp=0xc420026f50
os/signal.loop()
	/usr/lib/go/src/os/signal/signal_unix.go:22 +0x22 fp=0xc420026fc0 sp=0xc420026f80
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420026fc8 sp=0xc420026fc0
created by os/signal.init.1
	/usr/lib/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 5 [select]:
runtime.gopark(0xc7bbf0, 0x0, 0xc12003, 0x6, 0x18, 0x2)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc42003dc08 sp=0xc42003dbd8
runtime.selectgoImpl(0xc42003df18, 0x0, 0x18)
	/usr/lib/go/src/runtime/select.go:423 +0x11d9 fp=0xc42003de30 sp=0xc42003dc08
runtime.selectgo(0xc42003df18)
	/usr/lib/go/src/runtime/select.go:238 +0x1c fp=0xc42003de58 sp=0xc42003de30
net/http.(*persistConn).writeLoop(0xc420157700)
	/usr/lib/go/src/net/http/transport.go:1646 +0x3bd fp=0xc42003dfa8 sp=0xc42003de58
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42003dfb0 sp=0xc42003dfa8
created by net/http.(*Transport).dialConn
	/usr/lib/go/src/net/http/transport.go:1063 +0x50e

goroutine 22 [runnable]:
runtime.gopark(0xc7bbb8, 0x1019ee0, 0xc15fc8, 0xa, 0x7f7f00000019, 0x4)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420192678 sp=0xc420192648
runtime.goparkunlock(0x1019ee0, 0xc15fc8, 0xa, 0x19, 0x4)
	/usr/lib/go/src/runtime/proc.go:265 +0x5e fp=0xc4201926b8 sp=0xc420192678
runtime.semacquire(0x100d988, 0x100d800)
	/usr/lib/go/src/runtime/sema.go:111 +0x20d fp=0xc420192720 sp=0xc4201926b8
runtime.gcMarkDone()
	/usr/lib/go/src/runtime/mgc.go:1055 +0x36 fp=0xc420192748 sp=0xc420192720
runtime.gcBgMarkWorker(0xc42001b500)
	/usr/lib/go/src/runtime/mgc.go:1497 +0x2c2 fp=0xc4201927b8 sp=0xc420192748
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4201927c0 sp=0xc4201927b8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 21 [garbage collection]:
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_amd64.s:252 fp=0xc420192d28 sp=0xc420192d20
runtime.gcMarkTermination()
	/usr/lib/go/src/runtime/mgc.go:1220 +0x17b fp=0xc420192f20 sp=0xc420192d28
runtime.gcMarkDone()
	/usr/lib/go/src/runtime/mgc.go:1150 +0x21e fp=0xc420192f48 sp=0xc420192f20
runtime.gcBgMarkWorker(0xc42001a000)
	/usr/lib/go/src/runtime/mgc.go:1497 +0x2c2 fp=0xc420192fb8 sp=0xc420192f48
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420192fc0 sp=0xc420192fb8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 23 [GC worker (idle)]:
runtime.gopark(0xc7ba28, 0xc4201efbb0, 0xc1a50b, 0x10, 0x14, 0x0)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420027f48 sp=0xc420027f18
runtime.gcBgMarkWorker(0xc42001ca00)
	/usr/lib/go/src/runtime/mgc.go:1420 +0x108 fp=0xc420027fb8 sp=0xc420027f48
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420027fc0 sp=0xc420027fb8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 24 [GC worker (idle)]:
runtime.gopark(0xc7ba28, 0xc4201efbc0, 0xc1a50b, 0x10, 0x14, 0x0)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420028748 sp=0xc420028718
runtime.gcBgMarkWorker(0xc42001e000)
	/usr/lib/go/src/runtime/mgc.go:1420 +0x108 fp=0xc4200287b8 sp=0xc420028748
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200287c0 sp=0xc4200287b8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 25 [GC worker (idle)]:
runtime.gopark(0xc7ba28, 0xc4201efbd0, 0xc1a50b, 0x10, 0x14, 0x0)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420028f48 sp=0xc420028f18
runtime.gcBgMarkWorker(0xc42001f500)
	/usr/lib/go/src/runtime/mgc.go:1420 +0x108 fp=0xc420028fb8 sp=0xc420028f48
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420028fc0 sp=0xc420028fb8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 26 [GC worker (idle)]:
runtime.gopark(0xc7ba28, 0xc4201efbe0, 0xc1a50b, 0x10, 0x14, 0x0)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420029748 sp=0xc420029718
runtime.gcBgMarkWorker(0xc420020a00)
	/usr/lib/go/src/runtime/mgc.go:1420 +0x108 fp=0xc4200297b8 sp=0xc420029748
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200297c0 sp=0xc4200297b8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 27 [GC worker (idle)]:
runtime.gopark(0xc7ba28, 0xc4201efbf0, 0xc1a50b, 0x10, 0x14, 0x0)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc420029f48 sp=0xc420029f18
runtime.gcBgMarkWorker(0xc420022000)
	/usr/lib/go/src/runtime/mgc.go:1420 +0x108 fp=0xc420029fb8 sp=0xc420029f48
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420029fc0 sp=0xc420029fb8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 28 [GC worker (idle)]:
runtime.gopark(0xc7ba28, 0xc4201efc00, 0xc1a50b, 0x10, 0x14, 0x0)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc42018e748 sp=0xc42018e718
runtime.gcBgMarkWorker(0xc420023500)
	/usr/lib/go/src/runtime/mgc.go:1420 +0x108 fp=0xc42018e7b8 sp=0xc42018e748
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42018e7c0 sp=0xc42018e7b8
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go/src/runtime/mgc.go:1341 +0x98

goroutine 38 [syscall]:
runtime.notetsleepg(0x100d198, 0xdf842ce84, 0x1)
	/usr/lib/go/src/runtime/lock_futex.go:205 +0x42 fp=0xc420191f40 sp=0xc420191f10
runtime.timerproc()
	/usr/lib/go/src/runtime/time.go:209 +0x2eb fp=0xc420191fc0 sp=0xc420191f40
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420191fc8 sp=0xc420191fc0
created by runtime.addtimerLocked
	/usr/lib/go/src/runtime/time.go:116 +0xee

goroutine 4 [IO wait]:
runtime.gopark(0xc7bb70, 0x7f7fbff837e0, 0xc125cc, 0x7, 0x5a941b, 0x5)
	/usr/lib/go/src/runtime/proc.go:259 +0x13a fp=0xc42003b4e8 sp=0xc42003b4b8
runtime.netpollblock(0x7f7fbff837b8, 0x72, 0xb)
	/usr/lib/go/src/runtime/netpoll.go:341 +0xeb fp=0xc42003b530 sp=0xc42003b4e8
net.runtime_pollWait(0x7f7fbff837b8, 0x72, 0x3)
	/usr/lib/go/src/runtime/netpoll.go:160 +0x59 fp=0xc42003b558 sp=0xc42003b530
net.(*pollDesc).wait(0xc42024a220, 0x72, 0xc42003b5e8, 0xc4200740a0)
	/usr/lib/go/src/net/fd_poll_runtime.go:73 +0x38 fp=0xc42003b580 sp=0xc42003b558
net.(*pollDesc).waitRead(0xc42024a220, 0xfd2fe0, 0xc4200740a0)
	/usr/lib/go/src/net/fd_poll_runtime.go:78 +0x34 fp=0xc42003b5b0 sp=0xc42003b580
net.(*netFD).Read(0xc42024a1c0, 0xc420268000, 0x1000, 0x1000, 0x0, 0xfd2fe0, 0xc4200740a0)
	/usr/lib/go/src/net/fd_unix.go:243 +0x1a1 fp=0xc42003b620 sp=0xc42003b5b0
net.(*conn).Read(0xc420086018, 0xc420268000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/net.go:173 +0x70 fp=0xc42003b690 sp=0xc42003b620
net.(Conn).Read-fm(0xc420268000, 0x1000, 0x1000, 0xc42003b700, 0xc42000f000, 0x451450)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/http.go:68 +0x4d fp=0xc42003b6d8 sp=0xc42003b690
github.com/ncw/rclone/fs.(*timeoutConn).readOrWrite(0xc420172040, 0xc42003b770, 0xc420268000, 0x1000, 0x1000, 0xc42000f000, 0x411156, 0xc42003b778)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/http.go:58 +0x69 fp=0xc42003b730 sp=0xc42003b6d8
github.com/ncw/rclone/fs.(*timeoutConn).Read(0xc420172040, 0xc420268000, 0x1000, 0x1000, 0xbb0dc0, 0x0, 0x7f7fc0012dc0)
	/build/rclone/src/rclone/.gopath/src/github.com/ncw/rclone/fs/http.go:68 +0x8b fp=0xc42003b798 sp=0xc42003b730
crypto/tls.(*block).readFromUntil(0xc420162540, 0x7f7fc0012dc0, 0xc420172040, 0x5, 0xc420172040, 0xc42003b818)
	/usr/lib/go/src/crypto/tls/conn.go:476 +0x91 fp=0xc42003b7e0 sp=0xc42003b798
crypto/tls.(*Conn).readRecord(0xc420150000, 0xc7bc17, 0xc420150108, 0xc4200001a0)
	/usr/lib/go/src/crypto/tls/conn.go:578 +0xc4 fp=0xc42003ba28 sp=0xc42003b7e0
crypto/tls.(*Conn).Read(0xc420150000, 0xc420250000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/crypto/tls/conn.go:1113 +0x116 fp=0xc42003ba78 sp=0xc42003ba28
net/http.(*persistConn).Read(0xc420157700, 0xc420250000, 0x1000, 0x1000, 0x6c1420, 0xc42003bb58, 0x4045ed)
	/usr/lib/go/src/net/http/transport.go:1261 +0x154 fp=0xc42003baf0 sp=0xc42003ba78
bufio.(*Reader).fill(0xc420018de0)
	/usr/lib/go/src/bufio/bufio.go:97 +0x10c fp=0xc42003bb50 sp=0xc42003baf0
bufio.(*Reader).Peek(0xc420018de0, 0x1, 0xc42003bbbd, 0x1, 0x0, 0xc420018e40, 0x0)
	/usr/lib/go/src/bufio/bufio.go:129 +0x62 fp=0xc42003bb68 sp=0xc42003bb50
net/http.(*persistConn).readLoop(0xc420157700)
	/usr/lib/go/src/net/http/transport.go:1418 +0x1a1 fp=0xc42003bfa8 sp=0xc42003bb68
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42003bfb0 sp=0xc42003bfa8
created by net/http.(*Transport).dialConn
	/usr/lib/go/src/net/http/transport.go:1062 +0x4e9

my go env:

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/[redacted]/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build632310134=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
@Skdp

This comment has been minimized.

Copy link

commented Feb 27, 2017

---OT---

@cynecx Leave me your contact details(jid) to ... you know where ... Would love to chat! Now I have more time on my hands. Took a while to find you mate. -skid 😘

@cynecx

This comment has been minimized.

Copy link
Author

commented Feb 27, 2017

@Skdp Seriously, I have no idea who you are. What's your bitbucket id, so I can add you to a private repo. I think it's better if don't use this issue thread for having private conversations.

@aclements

This comment has been minimized.

Copy link
Member

commented Mar 1, 2017

Ah! I was getting thrown off because this isn't the local frame of database/sql.(*Rows).awaitDone, it's the arguments to database/sql.(*Rows).close (passed by awaitDone).

Click for database/sql.(*Rows).close disassembly
"".(*Rows).close t=1 size=635 args=0x28 locals=0x38
        0x0000 00000 (/go/src/database/sql/sql.go:2421)        TEXT    "".(*Rows).close(SB), $56-40
        0x0000 00000 (/go/src/database/sql/sql.go:2421)        MOVQ    (TLS), CX
        0x0009 00009 (/go/src/database/sql/sql.go:2421)        CMPQ    SP, 16(CX)
        0x000d 00013 (/go/src/database/sql/sql.go:2421)        JLS     625
        0x0013 00019 (/go/src/database/sql/sql.go:2421)        SUBQ    $56, SP
        0x0017 00023 (/go/src/database/sql/sql.go:2421)        MOVQ    BP, 48(SP)
        0x001c 00028 (/go/src/database/sql/sql.go:2421)        LEAQ    48(SP), BP
        0x0021 00033 (/go/src/database/sql/sql.go:2421)        FUNCDATA        $0, gclocals·6f2e5fbd0977091d169b23838d23bf
9c(SB)
        0x0021 00033 (/go/src/database/sql/sql.go:2421)        FUNCDATA        $1, gclocals·1bbed2bdcbe153739de5acc844c428
b7(SB)
        0x0021 00033 (/go/src/database/sql/sql.go:2421)        LEAQ    type.error(SB), AX
        0x0028 00040 (/go/src/database/sql/sql.go:2421)        MOVQ    AX, (SP)
        0x002c 00044 (/go/src/database/sql/sql.go:2421)        PCDATA  $0, $1
        0x002c 00044 (/go/src/database/sql/sql.go:2421)        CALL    runtime.newobject(SB)
        0x0031 00049 (/go/src/database/sql/sql.go:2421)        MOVQ    8(SP), AX
        0x0036 00054 (/go/src/database/sql/sql.go:2421)        MOVQ    AX, "".&err+40(SP)
        0x003b 00059 (/go/src/database/sql/sql.go:2421)        MOVQ    "".err+72(FP), CX
        0x0040 00064 (/go/src/database/sql/sql.go:2421)        MOVQ    CX, (AX)
        0x0043 00067 (/go/src/database/sql/sql.go:2421)        MOVL    runtime.writeBarrier(SB), CX
        0x0049 00073 (/go/src/database/sql/sql.go:2421)        LEAQ    8(AX), DX
        0x004d 00077 (/go/src/database/sql/sql.go:2421)        MOVQ    DX, ""..autotmp_1216+32(SP)
        0x0052 00082 (/go/src/database/sql/sql.go:2421)        TESTL   CX, CX
        0x0054 00084 (/go/src/database/sql/sql.go:2421)        JNE     $0, 591
        0x005a 00090 (/go/src/database/sql/sql.go:2421)        MOVQ    "".err+80(FP), CX
        0x005f 00095 (/go/src/database/sql/sql.go:2421)        MOVQ    CX, 8(AX)
        0x0063 00099 (/go/src/database/sql/sql.go:2421)        MOVQ    $0, "".~r1+88(FP)
        0x006c 00108 (/go/src/database/sql/sql.go:2421)        MOVQ    $0, "".~r1+96(FP)
        0x0075 00117 (/go/src/database/sql/sql.go:2422)        MOVQ    "".rs+64(FP), CX
        0x007a 00122 (/go/src/database/sql/sql.go:2422)        TESTB   AL, (CX)
        0x007c 00124 (/go/src/database/sql/sql.go:2422)        LEAQ    48(CX), BX
        0x0080 00128 (/go/src/database/sql/sql.go:2422)        MOVQ    BX, ""..autotmp_1217+24(SP)
        0x0085 00133 (/go/src/database/sql/sql.go:2422)        MOVQ    BX, (SP)
        0x0089 00137 (/go/src/database/sql/sql.go:2422)        PCDATA  $0, $3
        0x0089 00137 (/go/src/database/sql/sql.go:2422)        CALL    sync.(*RWMutex).Lock(SB)
        0x008e 00142 (/go/src/database/sql/sql.go:2423)        MOVQ    ""..autotmp_1217+24(SP), AX
        0x0093 00147 (/go/src/database/sql/sql.go:2423)        MOVQ    AX, 16(SP)
        0x0098 00152 (/go/src/database/sql/sql.go:2423)        MOVL    $8, (SP)
        0x009f 00159 (/go/src/database/sql/sql.go:2423)        LEAQ    sync.(*RWMutex).Unlock·f(SB), AX
        0x00a6 00166 (/go/src/database/sql/sql.go:2423)        MOVQ    AX, 8(SP)
        0x00ab 00171 (/go/src/database/sql/sql.go:2423)        PCDATA  $0, $2
        0x00ab 00171 (/go/src/database/sql/sql.go:2423)        CALL    runtime.deferproc(SB)
        0x00b0 00176 (/go/src/database/sql/sql.go:2423)        TESTL   AX, AX
        0x00b2 00178 (/go/src/database/sql/sql.go:2423)        JNE     575
        0x00b8 00184 (/go/src/database/sql/sql.go:2425)        MOVQ    "".rs+64(FP), AX
        0x00bd 00189 (/go/src/database/sql/sql.go:2425)        MOVBLZX 72(AX), CX
        0x00c1 00193 (/go/src/database/sql/sql.go:2425)        TESTB   CL, CL
        0x00c3 00195 (/go/src/database/sql/sql.go:2425)        JNE     $0, 541
        0x00c9 00201 (/go/src/database/sql/sql.go:2428)        MOVB    $1, 72(AX)
        0x00cd 00205 (/go/src/database/sql/sql.go:2430)        MOVQ    80(AX), CX
        0x00d1 00209 (/go/src/database/sql/sql.go:2430)        TESTQ   CX, CX
        0x00d4 00212 (/go/src/database/sql/sql.go:2430)        JEQ     473
        0x00da 00218 (/go/src/database/sql/sql.go:2434)        MOVQ    24(AX), CX
        0x00de 00222 (/go/src/database/sql/sql.go:2434)        MOVQ    16(AX), DX
        0x00e2 00226 (/go/src/database/sql/sql.go:2434)        MOVQ    32(DX), DX
        0x00e6 00230 (/go/src/database/sql/sql.go:2434)        MOVQ    CX, (SP)
        0x00ea 00234 (/go/src/database/sql/sql.go:2434)        PCDATA  $0, $2
        0x00ea 00234 (/go/src/database/sql/sql.go:2434)        CALL    DX
        0x00ec 00236 (/go/src/database/sql/sql.go:2434)        MOVQ    16(SP), AX
        0x00f1 00241 (/go/src/database/sql/sql.go:2434)        MOVQ    8(SP), CX
        0x00f6 00246 (/go/src/database/sql/sql.go:2434)        MOVQ    "".&err+40(SP), DX
        0x00fb 00251 (/go/src/database/sql/sql.go:2434)        MOVQ    CX, (DX)
        0x00fe 00254 (/go/src/database/sql/sql.go:2434)        MOVL    runtime.writeBarrier(SB), CX
        0x0104 00260 (/go/src/database/sql/sql.go:2434)        TESTL   CX, CX
        0x0106 00262 (/go/src/database/sql/sql.go:2434)        JNE     $0, 444
        0x010c 00268 (/go/src/database/sql/sql.go:2434)        MOVQ    AX, 8(DX)
        0x0110 00272 (/go/src/database/sql/sql.go:2435)        MOVQ    "".rowsCloseHook(SB), AX
        0x0117 00279 (/go/src/database/sql/sql.go:2435)        MOVQ    (AX), CX
        0x011a 00282 (/go/src/database/sql/sql.go:2435)        MOVQ    AX, DX
        0x011d 00285 (/go/src/database/sql/sql.go:2435)        PCDATA  $0, $4
        0x011d 00285 (/go/src/database/sql/sql.go:2435)        CALL    CX
        0x011f 00287 (/go/src/database/sql/sql.go:2435)        MOVQ    (SP), DX
        0x0123 00291 (/go/src/database/sql/sql.go:2435)        TESTQ   DX, DX
        0x0126 00294 (/go/src/database/sql/sql.go:2435)        JNE     $0, 415
        0x0128 00296 (/go/src/database/sql/sql.go:2438)        MOVQ    "".rs+64(FP), AX
        0x012d 00301 (/go/src/database/sql/sql.go:2438)        MOVQ    32(AX), DX
        0x0131 00305 (/go/src/database/sql/sql.go:2438)        TESTQ   DX, DX
        0x0134 00308 (/go/src/database/sql/sql.go:2438)        JNE     $0, 403
        0x0136 00310 (/go/src/database/sql/sql.go:2442)        MOVQ    40(AX), CX
        0x013a 00314 (/go/src/database/sql/sql.go:2442)        TESTQ   CX, CX
        0x013d 00317 (/go/src/database/sql/sql.go:2442)        JNE     $0, 387
        0x013f 00319 (/go/src/database/sql/sql.go:2445)        MOVQ    8(AX), DX
        0x0143 00323 (/go/src/database/sql/sql.go:2445)        MOVQ    "".&err+40(SP), AX
        0x0148 00328 (/go/src/database/sql/sql.go:2445)        MOVQ    8(AX), CX
        0x014c 00332 (/go/src/database/sql/sql.go:2445)        MOVQ    (AX), BX
        0x014f 00335 (/go/src/database/sql/sql.go:2445)        MOVQ    BX, (SP)
        0x0153 00339 (/go/src/database/sql/sql.go:2445)        MOVQ    CX, 8(SP)
        0x0158 00344 (/go/src/database/sql/sql.go:2445)        MOVQ    (DX), CX
        0x015b 00347 (/go/src/database/sql/sql.go:2445)        PCDATA  $0, $5
        0x015b 00347 (/go/src/database/sql/sql.go:2445)        CALL    CX
        0x015d 00349 (/go/src/database/sql/sql.go:2446)        MOVQ    "".&err+40(SP), AX
        0x0162 00354 (/go/src/database/sql/sql.go:2446)        MOVQ    8(AX), CX
        0x0166 00358 (/go/src/database/sql/sql.go:2446)        MOVQ    (AX), AX
        0x0169 00361 (/go/src/database/sql/sql.go:2446)        MOVQ    AX, "".~r1+88(FP)
        0x016e 00366 (/go/src/database/sql/sql.go:2446)        MOVQ    CX, "".~r1+96(FP)
        0x0173 00371 (/go/src/database/sql/sql.go:2446)        PCDATA  $0, $6
        0x0173 00371 (/go/src/database/sql/sql.go:2446)        XCHGL   AX, AX
        0x0174 00372 (/go/src/database/sql/sql.go:2446)        CALL    runtime.deferreturn(SB)
        0x0179 00377 (/go/src/database/sql/sql.go:2446)        MOVQ    48(SP), BP
        0x017e 00382 (/go/src/database/sql/sql.go:2446)        ADDQ    $56, SP
        0x0182 00386 (/go/src/database/sql/sql.go:2446)        RET
        0x0183 00387 (/go/src/database/sql/sql.go:2443)        MOVQ    CX, (SP)
        0x0187 00391 (/go/src/database/sql/sql.go:2443)        PCDATA  $0, $4
        0x0187 00391 (/go/src/database/sql/sql.go:2443)        CALL    "".(*driverStmt).Close(SB)
        0x018c 00396 (/go/src/database/sql/sql.go:2445)        MOVQ    "".rs+64(FP), AX
        0x0191 00401 (/go/src/database/sql/sql.go:2442)        JMP     319
        0x0193 00403 (/go/src/database/sql/sql.go:2439)        MOVQ    (DX), CX
        0x0196 00406 (/go/src/database/sql/sql.go:2439)        PCDATA  $0, $4
        0x0196 00406 (/go/src/database/sql/sql.go:2439)        CALL    CX
        0x0198 00408 (/go/src/database/sql/sql.go:2442)        MOVQ    "".rs+64(FP), AX
        0x019d 00413 (/go/src/database/sql/sql.go:2438)        JMP     310
        0x019f 00415 (/go/src/database/sql/sql.go:2436)        MOVQ    "".&err+40(SP), AX
        0x01a4 00420 (/go/src/database/sql/sql.go:2436)        MOVQ    AX, 8(SP)
        0x01a9 00425 (/go/src/database/sql/sql.go:2436)        MOVQ    "".rs+64(FP), CX
        0x01ae 00430 (/go/src/database/sql/sql.go:2436)        MOVQ    CX, (SP)
        0x01b2 00434 (/go/src/database/sql/sql.go:2436)        MOVQ    (DX), BX
        0x01b5 00437 (/go/src/database/sql/sql.go:2436)        PCDATA  $0, $4
        0x01b5 00437 (/go/src/database/sql/sql.go:2436)        CALL    BX
        0x01b7 00439 (/go/src/database/sql/sql.go:2435)        JMP     296
        0x01bc 00444 (/go/src/database/sql/sql.go:2434)        MOVQ    ""..autotmp_1216+32(SP), CX
        0x01c1 00449 (/go/src/database/sql/sql.go:2434)        MOVQ    CX, (SP)
        0x01c5 00453 (/go/src/database/sql/sql.go:2434)        MOVQ    AX, 8(SP)
        0x01ca 00458 (/go/src/database/sql/sql.go:2434)        PCDATA  $0, $4
        0x01ca 00458 (/go/src/database/sql/sql.go:2434)        CALL    runtime.writebarrierptr(SB)
        0x01cf 00463 (/go/src/database/sql/sql.go:2445)        MOVQ    "".&err+40(SP), DX
        0x01d4 00468 (/go/src/database/sql/sql.go:2434)        JMP     272
        0x01d9 00473 (/go/src/database/sql/sql.go:2431)        MOVQ    "".&err+40(SP), CX
        0x01de 00478 (/go/src/database/sql/sql.go:2431)        MOVQ    8(CX), DX
        0x01e2 00482 (/go/src/database/sql/sql.go:2431)        MOVQ    (CX), BX
        0x01e5 00485 (/go/src/database/sql/sql.go:2431)        MOVQ    BX, 80(AX)
        0x01e9 00489 (/go/src/database/sql/sql.go:2431)        MOVL    runtime.writeBarrier(SB), BX
        0x01ef 00495 (/go/src/database/sql/sql.go:2431)        LEAQ    88(AX), SI
        0x01f3 00499 (/go/src/database/sql/sql.go:2431)        TESTL   BX, BX
        0x01f5 00501 (/go/src/database/sql/sql.go:2431)        JNE     $0, 512
        0x01f7 00503 (/go/src/database/sql/sql.go:2431)        MOVQ    DX, 88(AX)
        0x01fb 00507 (/go/src/database/sql/sql.go:2431)        JMP     218
        0x0200 00512 (/go/src/database/sql/sql.go:2431)        MOVQ    SI, (SP)
        0x0204 00516 (/go/src/database/sql/sql.go:2431)        MOVQ    DX, 8(SP)
        0x0209 00521 (/go/src/database/sql/sql.go:2431)        PCDATA  $0, $2
        0x0209 00521 (/go/src/database/sql/sql.go:2431)        CALL    runtime.writebarrierptr(SB)
        0x020e 00526 (/go/src/database/sql/sql.go:2434)        MOVQ    "".rs+64(FP), AX
        0x0213 00531 (/go/src/database/sql/sql.go:2434)        MOVQ    "".&err+40(SP), CX
        0x0218 00536 (/go/src/database/sql/sql.go:2431)        JMP     218
        0x021d 00541 (/go/src/database/sql/sql.go:2426)        MOVQ    $0, "".~r1+88(FP)
        0x0226 00550 (/go/src/database/sql/sql.go:2426)        MOVQ    $0, "".~r1+96(FP)
        0x022f 00559 (/go/src/database/sql/sql.go:2426)        PCDATA  $0, $6
        0x022f 00559 (/go/src/database/sql/sql.go:2426)        XCHGL   AX, AX
        0x0230 00560 (/go/src/database/sql/sql.go:2426)        CALL    runtime.deferreturn(SB)
        0x0235 00565 (/go/src/database/sql/sql.go:2426)        MOVQ    48(SP), BP
        0x023a 00570 (/go/src/database/sql/sql.go:2426)        ADDQ    $56, SP
        0x023e 00574 (/go/src/database/sql/sql.go:2426)        RET
        0x023f 00575 (/go/src/database/sql/sql.go:2423)        PCDATA  $0, $6
        0x023f 00575 (/go/src/database/sql/sql.go:2423)        XCHGL   AX, AX
        0x0240 00576 (/go/src/database/sql/sql.go:2423)        CALL    runtime.deferreturn(SB)
        0x0245 00581 (/go/src/database/sql/sql.go:2423)        MOVQ    48(SP), BP
        0x024a 00586 (/go/src/database/sql/sql.go:2423)        ADDQ    $56, SP
        0x024e 00590 (/go/src/database/sql/sql.go:2423)        RET
        0x024f 00591 (/go/src/database/sql/sql.go:2421)        MOVQ    DX, (SP)
        0x0253 00595 (/go/src/database/sql/sql.go:2421)        MOVQ    "".err+80(FP), CX
        0x0258 00600 (/go/src/database/sql/sql.go:2421)        MOVQ    CX, 8(SP)
        0x025d 00605 (/go/src/database/sql/sql.go:2421)        PCDATA  $0, $2
        0x025d 00605 (/go/src/database/sql/sql.go:2421)        CALL    runtime.writebarrierptr(SB)
        0x0262 00610 (/go/src/database/sql/sql.go:2431)        MOVQ    "".&err+40(SP), AX
        0x0267 00615 (/go/src/database/sql/sql.go:2434)        MOVQ    ""..autotmp_1216+32(SP), DX
        0x026c 00620 (/go/src/database/sql/sql.go:2421)        JMP     99
        0x0271 00625 (/go/src/database/sql/sql.go:2421)        NOP
        0x0271 00625 (/go/src/database/sql/sql.go:2421)        PCDATA  $0, $-1
        0x0271 00625 (/go/src/database/sql/sql.go:2421)        CALL    runtime.morestack_noctxt(SB)
        0x0276 00630 (/go/src/database/sql/sql.go:2421)        JMP     0
        0x0000 64 48 8b 0c 25 00 00 00 00 48 3b 61 10 0f 86 5e  dH..%....H;a...^
        0x0010 02 00 00 48 83 ec 38 48 89 6c 24 30 48 8d 6c 24  ...H..8H.l$0H.l$
        0x0020 30 48 8d 05 00 00 00 00 48 89 04 24 e8 00 00 00  0H......H..$....
        0x0030 00 48 8b 44 24 08 48 89 44 24 28 48 8b 4c 24 48  .H.D$.H.D$(H.L$H
        0x0040 48 89 08 8b 0d 00 00 00 00 48 8d 50 08 48 89 54  H........H.P.H.T
        0x0050 24 20 85 c9 0f 85 f5 01 00 00 48 8b 4c 24 50 48  $ ........H.L$PH
        0x0060 89 48 08 48 c7 44 24 58 00 00 00 00 48 c7 44 24  .H.H.D$X....H.D$
        0x0070 60 00 00 00 00 48 8b 4c 24 40 84 01 48 8d 59 30  `....H.L$@..H.Y0
        0x0080 48 89 5c 24 18 48 89 1c 24 e8 00 00 00 00 48 8b  H.\$.H..$.....H.
        0x0090 44 24 18 48 89 44 24 10 c7 04 24 08 00 00 00 48  D$.H.D$...$....H
        0x00a0 8d 05 00 00 00 00 48 89 44 24 08 e8 00 00 00 00  ......H.D$......
        0x00b0 85 c0 0f 85 87 01 00 00 48 8b 44 24 40 0f b6 48  ........H.D$@..H
        0x00c0 48 84 c9 0f 85 54 01 00 00 c6 40 48 01 48 8b 48  H....T....@H.H.H
        0x00d0 50 48 85 c9 0f 84 ff 00 00 00 48 8b 48 18 48 8b  PH........H.H.H.
        0x00e0 50 10 48 8b 52 20 48 89 0c 24 ff d2 48 8b 44 24  P.H.R H..$..H.D$
        0x00f0 10 48 8b 4c 24 08 48 8b 54 24 28 48 89 0a 8b 0d  .H.L$.H.T$(H....
        0x0100 00 00 00 00 85 c9 0f 85 b0 00 00 00 48 89 42 08  ............H.B.
        0x0110 48 8b 05 00 00 00 00 48 8b 08 48 89 c2 ff d1 48  H......H..H....H
        0x0120 8b 14 24 48 85 d2 75 77 48 8b 44 24 40 48 8b 50  ..$H..uwH.D$@H.P
        0x0130 20 48 85 d2 75 5d 48 8b 48 28 48 85 c9 75 44 48   H..u]H.H(H..uDH
        0x0140 8b 50 08 48 8b 44 24 28 48 8b 48 08 48 8b 18 48  .P.H.D$(H.H.H..H
        0x0150 89 1c 24 48 89 4c 24 08 48 8b 0a ff d1 48 8b 44  ..$H.L$.H....H.D
        0x0160 24 28 48 8b 48 08 48 8b 00 48 89 44 24 58 48 89  $(H.H.H..H.D$XH.
        0x0170 4c 24 60 90 e8 00 00 00 00 48 8b 6c 24 30 48 83  L$`......H.l$0H.
        0x0180 c4 38 c3 48 89 0c 24 e8 00 00 00 00 48 8b 44 24  .8.H..$.....H.D$
        0x0190 40 eb ac 48 8b 0a ff d1 48 8b 44 24 40 eb 97 48  @..H....H.D$@..H
        0x01a0 8b 44 24 28 48 89 44 24 08 48 8b 4c 24 40 48 89  .D$(H.D$.H.L$@H.
        0x01b0 0c 24 48 8b 1a ff d3 e9 6c ff ff ff 48 8b 4c 24  .$H.....l...H.L$
        0x01c0 20 48 89 0c 24 48 89 44 24 08 e8 00 00 00 00 48   H..$H.D$......H
        0x01d0 8b 54 24 28 e9 37 ff ff ff 48 8b 4c 24 28 48 8b  .T$(.7...H.L$(H.
        0x01e0 51 08 48 8b 19 48 89 58 50 8b 1d 00 00 00 00 48  Q.H..H.XP......H
        0x01f0 8d 70 58 85 db 75 09 48 89 50 58 e9 da fe ff ff  .pX..u.H.PX.....
        0x0200 48 89 34 24 48 89 54 24 08 e8 00 00 00 00 48 8b  H.4$H.T$......H.
        0x0210 44 24 40 48 8b 4c 24 28 e9 bd fe ff ff 48 c7 44  D$@H.L$(.....H.D
        0x0220 24 58 00 00 00 00 48 c7 44 24 60 00 00 00 00 90  $X....H.D$`.....
        0x0230 e8 00 00 00 00 48 8b 6c 24 30 48 83 c4 38 c3 90  .....H.l$0H..8..
        0x0240 e8 00 00 00 00 48 8b 6c 24 30 48 83 c4 38 c3 48  .....H.l$0H..8.H
        0x0250 89 14 24 48 8b 4c 24 50 48 89 4c 24 08 e8 00 00  ..$H.L$PH.L$....
        0x0260 00 00 48 8b 44 24 28 48 8b 54 24 20 e9 f2 fd ff  ..H.D$(H.T$ ....
        0x0270 ff e8 00 00 00 00 e9 85 fd ff ff                 ...........
        rel 5+4 t=16 TLS+0
        rel 36+4 t=15 type.error+0
        rel 45+4 t=8 runtime.newobject+0
        rel 69+4 t=15 runtime.writeBarrier+0
        rel 138+4 t=8 sync.(*RWMutex).Lock+0
        rel 162+4 t=15 sync.(*RWMutex).Unlock·f+0
        rel 172+4 t=8 runtime.deferproc+0
        rel 234+0 t=11 +0
        rel 256+4 t=15 runtime.writeBarrier+0
        rel 275+4 t=15 "".rowsCloseHook+0
        rel 285+0 t=11 +0
        rel 347+0 t=11 +0
        rel 373+4 t=8 runtime.deferreturn+0
        rel 392+4 t=8 "".(*driverStmt).Close+0
        rel 406+0 t=11 +0
        rel 437+0 t=11 +0
        rel 459+4 t=8 runtime.writebarrierptr+0
        rel 491+4 t=15 runtime.writeBarrier+0
        rel 522+4 t=8 runtime.writebarrierptr+0
        rel 561+4 t=8 runtime.deferreturn+0
        rel 577+4 t=8 runtime.deferreturn+0
        rel 606+4 t=8 runtime.writebarrierptr+0
        rel 626+4 t=8 runtime.morestack_noctxt+0
gclocals·6f2e5fbd0977091d169b23838d23bf9c t=8 dupok size=15
        0x0000 07 00 00 00 05 00 00 00 07 1f 19 19 19 18 18     ...............
gclocals·1bbed2bdcbe153739de5acc844c428b7 t=8 dupok size=15
        0x0000 07 00 00 00 03 00 00 00 00 00 06 07 04 04 00     ...............

The decoded argument map is: [0b00111, 0b11111, 0b11001, 0b11001, 0b11001, 0b11000, 0b11000].

The crashing return PC is +0x31 (we're in the call to newobject right at the beginning). At this PC we're on argument map entry 1, in which all argument words are live pointers.

The last two words of the argument frame (where we're seeing checkmark failures) are the error result. I believe the problem is that nothing zeroed these words at this point. They either need to be zeroed or marked not-live. @khr @drchase

@randall77

This comment has been minimized.

Copy link
Contributor

commented Mar 1, 2017

So then this is a dup of #19078?

@aclements

This comment has been minimized.

Copy link
Member

commented Mar 1, 2017

So then this is a dup of #19078?

That seems likely for the database/sql trybot failure. The other failures were reported on Go 1.7.5 (#19029 (comment)) and Go 1.8rc3 (#19029 (comment)), so it's probably a different failure.

@aclements

This comment has been minimized.

Copy link
Member

commented Mar 1, 2017

@greyltc, which exact rclone commit did you get that failure at? Also, it would be handy if you posted a few more failures (at the same commit or a different commit; just let us know).

@greyltc

This comment has been minimized.

Copy link

commented Mar 2, 2017

@aclements, this was with the 1.35 release, ncw/rclone@5b8b379

here are five more traces: https://gist.github.com/greyltc/5a75b4bdcf8538c54e53472c857b17a9

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2017

@greyltc, can you can gather 5 more traces with GODEBUG=gccheckmark=1 ?

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2017

@randall77, any ETA on fixing #19078? It would be nice to eliminate that from future debugging considerations.

@greyltc

This comment has been minimized.

Copy link

commented Mar 2, 2017

@rsc, here are those five additional traces, this time with GODEBUG=gccheckmark=1

https://gist.github.com/greyltc/8cc55ce7c1d9fa00b065f0bfbbf77b29

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2017

Thanks. The unscanned pointers are all over the place, not just in the same data structure at the same position each time. That's useful data, although still not conclusive.

@randall77

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2017

@rsc: nothing ready yet. I may have a fix by this weekend.

@aclements

This comment has been minimized.

Copy link
Member

commented Mar 3, 2017

For completeness, I analyzed @greyltc's five new traces:

fail_10.txt: Clearly still crypto/x509.Certificate, though this time it's the Subject.CommonName field.
fail_6.txt, fail_7.txt: An array or backing store for a slice of pointers. There isn't enough information to figure out what they're pointing to.
fail_8.txt: Another crypto/x509.Certificate, this time the Extensions slice base pointer.
fail_9.txt: This doesn't seem to match anything in the binary.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 3, 2017

@aclements, fail_9 looks like an array of (slice+interface) pairs, perhaps length 3, or perhaps 4 with a final zeroed pair.

@greyltc

This comment has been minimized.

Copy link

commented Mar 7, 2017

Would it be helpful for me to gather any more failure logs?

@adamflott

This comment has been minimized.

Copy link

commented Mar 7, 2017

I've been getting this crash with upgrading to 1.7.4 (won't be able to use 1.7.5) with cgo and leveldb possibly playing into it. I'm not sure what actions trigger, but its happening routinely in our testing environment.

golang_sweep_increase.txt

@aclements

This comment has been minimized.

Copy link
Member

commented Mar 8, 2017

@greyltc, unfortunately more logs probably wouldn't help at this point. The logs you sent don't fit into any of the patterns we'd expect for this failure, so we're a bit stumped. Do you (or @ncw) know if other rclone users are having similar problems? I'm wondering if it may be an environment issue in your case.

@adamflott, I assume your code isn't public? Can you try reproducing it a few times with GODEBUG=gccheckmark=1 GOTRACEBACK=crash? It's also possible that setting a lower GOGC value (e.g., GOGC=10) will cause it to happen more often.

@ncw

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2017

@aclements I haven't had any other reports of that bug and I haven't seen it myself in testing :-(

@greyltc

This comment has been minimized.

Copy link

commented Mar 13, 2017

@ncw try these steps to repeat it:

  1. be using official rclone package in Arch Linux "rclone-1.35"
  2. use rclone config to create a google drive remote
  3. provide your own oauth client id and secret from the google dev api
  4. use rclone config to create an encrypted remote backed by the google drive remote you just created
  5. use rclone mount to mount the encrypted remote
  6. copy large data in and out of the mount point
@adamflott

This comment has been minimized.

Copy link

commented Mar 20, 2017

@aclements code is not public. With those env vars I get an instant crash on startup, although not always. However I've been running a build with 1.6.3 and I have not seen a crash over a few days.

gp 0xc42006ad00 goid 34 status 3 gcscandone false gcscanvalid true
fatal error: scan missed a g

runtime stack:
runtime.throw(0xa78c07, 0xf)
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/panic.go:566 +0x95 fp=0x7f24e7f17d50 sp=0x7f24e7f17d30
runtime.gcMarkRootCheck()
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/mgcmark.go:117 +0x20b fp=0x7f24e7f17da0 sp=0x7f24e7f17d50
runtime.gcMark(0x9bd1f5f9d3035)
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/mgc.go:1583 +0x49c fp=0x7f24e7f17de8 sp=0x7f24e7f17da0
runtime.gcMarkTermination.func1()
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/mgc.go:1178 +0x2d fp=0x7f24e7f17e00 sp=0x7f24e7f17de8
runtime.systemstack(0xc420026a00)
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/asm_amd64.s:298 +0x79 fp=0x7f24e7f17e08 sp=0x7f24e7f17e00
runtime.mstart()
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/proc.go:1079 fp=0x7f24e7f17e10 sp=0x7f24e7f17e08

goroutine 20 [garbage collection]:
runtime.systemstack_switch()
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/asm_amd64.s:252 fp=0xc42002d528 sp=0xc42002d520
runtime.gcMarkTermination()
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/mgc.go:1186 +0x14e fp=0xc42002d720 sp=0xc42002d528
runtime.gcMarkDone()
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/mgc.go:1150 +0x21e fp=0xc42002d748 sp=0xc42002d720
runtime.gcBgMarkWorker(0xc420028000)
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/mgc.go:1497 +0x2c2 fp=0xc42002d7b8 sp=0xc42002d748
runtime.goexit()
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42002d7c0 sp=0xc42002d7b8
created by runtime.gcBgMarkStartWorkers
        /ghostcache/nightly_build/components/tablepusher/akamai/go/src/runtime/mgc.go:1341 +0x98

@aclements aclements modified the milestones: Go1.9Maybe, Go1.8Maybe May 18, 2017

@aclements

This comment has been minimized.

Copy link
Member

commented Jul 18, 2017

@ncw, @adamflott, same question for both of you: are you still able to reproduce this on Go 1.8 or current master (soon to be 1.9)?

I'm particularly wondering if commit c1730ae (released in Go 1.8) fixed this.

@ncw

This comment has been minimized.

Copy link
Contributor

commented Jul 19, 2017

@aclements I never managed to reproduce this and I haven't had any other reports of it.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2017

Rereading this issue it seems to combine several different problems, some clearly fixed, some possibly fixed, none clearly still relevant. It's gotten hard to pull out the specific information for a specific problem. Therefore, I'm going to close this issue as at least partially fixed. If you can still produce a problem when using Go1.9rc1 or later, please open a new issue that references this one and provides the details for your problem. Thanks.

@bozaro

This comment has been minimized.

Copy link

commented Aug 31, 2017

It's looks like a bug #21717

@aclements

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

It's looks like a bug #21717

"Sweep increased allocation count" means "something went wrong somewhere in garbage collector or maybe the runtime or maybe unsafe code", so the similar panic is not a good indicator that it's the same bug. :)

@golang golang locked and limited conversation to collaborators Aug 31, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.