cmd/gc: fatal error: bad pointer in write barrier

[davecheney]

fatal error: bad pointer in write barrier

runtime stack:
runtime.throw(0x18cb150, 0x1c)
        /home/dfc/go/src/runtime/panic.go:527 +0x96
runtime.writebarrierptr_nostore.func1()
        /home/dfc/go/src/runtime/mbarrier.go:125 +0x33
runtime.systemstack(0xc20801e000)
        /home/dfc/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
        /home/dfc/go/src/runtime/proc1.go:682

goroutine 9999 [running]:
runtime.systemstack_switch()
        /home/dfc/go/src/runtime/asm_amd64.s:216 fp=0xc2086b37f8 sp=0xc2086b37f0
runtime.writebarrierptr_nostore(0xc20878a3a0, 0xb)
        /home/dfc/go/src/runtime/mbarrier.go:125 +0x5c fp=0xc2086b3810 sp=0xc2086b37f8
runtime.heapBitsBulkBarrier(0xc20878a358, 0x50)
        /home/dfc/go/src/runtime/mbitmap.go:347 +0x189 fp=0xc2086b3860 sp=0xc2086b3810
runtime.callwritebarrier(0xc208356820, 0xc20878a320, 0x50, 0x38)
        /home/dfc/go/src/runtime/mbarrier.go:198 +0xf1 fp=0xc2086b3878 sp=0xc2086b3860
runtime.call128(0xc208356820, 0x1614320, 0xc20878a320, 0x3800000050)
        /home/dfc/go/src/runtime/asm_amd64.s:423 +0xae fp=0xc2086b3900 sp=0xc2086b3878
reflect.Value.call(0x1614220, 0xc208556240, 0x513, 0x1780a90, 0x4, 0xc208c9ac20, 0x1, 0x1, 0x0, 0x0, ...)
        /home/dfc/go/src/reflect/value.go:432 +0x120d fp=0xc2086b3c50 sp=0xc2086b3900
reflect.Value.Call(0x1614220, 0xc208556240, 0x513, 0xc208c9ac20, 0x1, 0x1, 0x0, 0x0, 0x0)
        /home/dfc/go/src/reflect/value.go:300 +0xb4 fp=0xc2086b3cb0 sp=0xc2086b3c50
github.com/juju/juju/rpc/rpcreflect.newMethod.func6(0x1614220, 0xc208556240, 0x16, 0x1559180, 0xc208d403c0, 0xd9, 0x0, 0x0, 0x0, 0x0, ...)
        /home/dfc/src/github.com/juju/juju/rpc/rpcreflect/type.go:323 +0x14e fp=0xc2086b3d70 sp=0xc2086b3cb0
github.com/juju/juju/apiserver.(*srvCaller).Call(0xc2087e8b40, 0x0, 0x0, 0x1559180, 0xc208d403c0, 0xd9, 0x0, 0x0, 0x0, 0x0, ...)
        /home/dfc/src/github.com/juju/juju/apiserver/root.go:131 +0x123 fp=0xc2086b3df8 sp=0xc2086b3d70
github.com/juju/juju/rpc.(*Conn).runRequest(0xc2089d23c0, 0x7f0766ece1b8, 0xc2087e8b40, 0x1b48690, 0x5c, 0xc208a30790, 0x7, 0x0, 0x0, 0x0, ...)
        /home/dfc/src/github.com/juju/juju/rpc/server.go:552 +0xd1 fp=0xc2086b3f30 sp=0xc2086b3df8
runtime.goexit()
        /home/dfc/go/src/runtime/asm_amd64.s:1670 +0x1 fp=0xc2086b3f38 sp=0xc2086b3f30
created by github.com/juju/juju/rpc.(*Conn).handleRequest
        /home/dfc/src/github.com/juju/juju/rpc/server.go:481 +0x717

go version devel +647026a Wed May 13 21:31:03 2015 +0000 linux/amd64

This is a bug turned up running the juju unit tests, so reproduction is convoluted. I can provide ssh access to a pre created environment if it useful.

cc @rsc @aclements

[bradfitz]

Any unsafe chicanery involved?

[davecheney]

nope.

% go list -f '{{.Imports}}' github.com/juju/juju/rpc github.com/juju/juju/apiserver github.com/juju/juju/rpc/rpcreflect | grep unsafe -c
0

[rsc]

I suspect this was fixed by https://go-review.googlesource.com/9888. The reflect package was not updated correctly for 1-bit stack maps until that CL. Please test again at that revision or later. Thanks.

[davecheney]

@rsc I've just checked with

go version devel +2b063bd Mon May 18 20:16:14 2015 +0000 linux/amd64

and the problem still exists. It is completely repeatable, but needs the full Juju song and dance environment to run the test case. If I set up a machine with all the necessary bits if that will help.

[rsc]

Yes please.

[davecheney]

After bisecting, I suspect the problem may have been introduced in 4212a3c

[davecheney]

I'm embarrassed to report that this test triggers multiple data races and
is probably what is causing this panic. http://paste.ubuntu.com/11235945/

I need to do some work to get my house in order before I can start throwing
stones at the runtime.

I'm sorry for wasting everyone's time.

On Wed, May 20, 2015 at 2:08 AM, Russ Cox notifications@github.com wrote:

Yes please.

—
Reply to this email directly or view it on GitHub
#10844 (comment).

[rsc]

There's still a real runtime bug here. On it. Thanks for the test case and test machine.