runtime: bad pointer 0xc900000000 in unexpected span (dup #13160?)

[fluhus]

I am getting this mysterious error message after my program runs for 2-3 hours (full stack trace below). The program runs on tons of data as a part of my academic research, so unfortunately I can't provide the data for you to reproduce it.

The code is here:
https://github.com/fluhus/prices/tree/master/src/items2
(sorry for scrappiness; It's in the middle of a rework)

More info

• See thread on golang nuts here: https://groups.google.com/forum/#!topic/golang-nuts/yn6_MDRFGr4
• golang 1.5.1 linux/amd64
• Machine runs Linux 64 bit with 32 cores and 64 GB RAM.
• No cgo in my code.
• No findings with race detector.

Let me know if you need additional information.

Thanks a lot!

runtime:objectstart Span weird: p=0xc900000000 k=0x6480000 s.start=0xc8fffe8000 s.limit=0xc900001fe0 s.state=2
fatal error: objectstart: bad pointer in unexpected span

runtime stack:
runtime.throw(0x65c4c0, 0x2b)
        /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.heapBitsForObject(0xc900000000, 0x0, 0x0, 0xc800000000, 0x7fad6c9123e0)
        /usr/local/go/src/runtime/mbitmap.go:217 +0x287
runtime.scanobject(0xc8ccd33b00, 0xc820036720)
        /usr/local/go/src/runtime/mgcmark.go:878 +0x239
runtime.gcDrainN(0xc820036720, 0x485e5)
        /usr/local/go/src/runtime/mgcmark.go:779 +0x8e
runtime.gcAssistAlloc.func1()
        /usr/local/go/src/runtime/mgcmark.go:232 +0x172
runtime.systemstack(0xc820022a00)
        /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
        /usr/local/go/src/runtime/proc1.go:674

[ianlancetaylor]

CC @RLH @aclements

See also #12157.

[fluhus]

I just tried running it using go 1.4.3 and it worked perfectly on the same machine.

[otoolep]

This issue has forced us to roll back building InfluxDB 0.9.5 with Go 1.5.1. We've reverted to Go 1.4.2 for the 0.9.5 release.

We are quite disappointed the 0.9.5 release will not be able to use the improved GC. Hopefully we'll have more luck with 0.9.6.

[RLH]

Any clues at how to reproduce this would be appreciated. You have already
indicated that there is no use of CGO/C and that the race detector reveals
no problem. This is good.

Is there any use of the unsafe package in the code?

If not then the next step is to work on reproducing the problem.

On Wed, Nov 11, 2015 at 6:53 PM, otoolep notifications@github.com wrote:

This issue has forced us to roll back building InfluxDB 0.9.5 with Go
1.5.1. We've reverted to Go 1.4.2 for this 0.9.5 release.

We are quite disappointed the 0.9.5 release will not be able to use the
improved GC. Hopefully we'll have more luck with 0.9.6.

—
Reply to this email directly or view it on GitHub
#13176 (comment).

[otoolep]

We don't have specific repro steps unfortunately. The problems were independently reported by two users in the InfluxDB community, who had their systems panic on them during normal usage, with this error.

[aclements]

Given the number of zeroes in the bad pointer p=0xc900000000, I strongly suspect this is the same as issue #13160. @fluhus, is it possible for you to try your program with current Go master, or with 4b7d5f0 cherry-picked on to the release-branch.go1.5 branch?

Also, if you need a workaround, set the GODEBUG=invalidptr=0 environment variable. (But obviously we'd like to fix this issue rather than work around it. :)

[fluhus]

@RLH No use of unsafe in my code. I am trying to find the exact input that would make it crash (though this might be gigs of data).

@aclements Will try and update on results. Thanks!

[aclements]

@fluhus, it's most likely not a particular input that makes it crash. The underlying bug is a concurrency issue between the garbage collector and copying/zeroing structures that have a pointer field at the end, so you just have to be unlucky. Thanks for working on reproducing it!

[RLH]

I would try Austin's suggestions before working on a repo....

On Thu, Nov 12, 2015 at 1:14 PM, Amit Lavon notifications@github.com
wrote:

@RLH https://github.com/RLH No use of unsafe in my code. I am trying to
find the exact input that would make it crash (though this might be gigs of
data).

@aclements https://github.com/aclements Will try and update on results.
Thanks!

—
Reply to this email directly or view it on GitHub
#13176 (comment).

[davecheney]

Would reducing GOGC below 100 increase the chance of reproducing the issue ?

On Fri, Nov 13, 2015 at 5:31 AM, Austin Clements notifications@github.com
wrote:

@fluhus https://github.com/fluhus, it's most likely not a particular
input that makes it crash. The underlying bug is a concurrency issue
between the garbage collector and copying/zeroing structures that have a
pointer field at the end, so you just have to be unlucky. Thanks for
working on reproducing it!

—
Reply to this email directly or view it on GitHub
#13176 (comment).

[aclements]

Would reducing GOGC below 100 increase the chance of reproducing the issue ?

Hmm. Probably, though I don't have an intuition for how much. Of course, if you set it too low it may perturb things enough to make the failure less likely.

[rsc]

I'm almost certain this is a duplicate of #13160, for the reasons Austin gives (low zeros in pointer).

[rsc]

Closing as duplicate of #13160.