runtime: Invalid heap pointer error in init() on Windows only using Go 1.4.2

[leothekim]

We are seeing this error on Windows 2008r2 64-bit when running our tests:

runtime: garbage collector found invalid heap pointer *(0xd74360+0x4a8)=0x9 s=nil
fatal error: invalid heap pointer

runtime stack:
runtime.throw(0xd66643)
  c:/go/src/runtime/panic.go:491 +0xad fp=0x6fb90 sp=0x6fb60
scanblock(0xd74360, 0xe3e0, 0x22c4f8)
  c:/go/src/runtime/mgc0.c:381 +0x558 fp=0x6fcd0 sp=0x6fb90
markroot(0xc0820020e0, 0x1)
  c:/go/src/runtime/mgc0.c:499 +0x170 fp=0x6fd30 sp=0x6fcd0
runtime.parfordo(0xc0820020e0)
  c:/go/src/runtime/parfor.c:76 +0xb9 fp=0x6fdb0 sp=0x6fd30
gc(0x6fee8)
  c:/go/src/runtime/mgc0.c:1442 +0x26c fp=0x6fec8 sp=0x6fdb0
runtime.gc_m()
  c:/go/src/runtime/mgc0.c:1371 +0x103 fp=0x6ff00 sp=0x6fec8
runtime.onM(0xd76e90)
  c:/go/src/runtime/asm_amd64.s:257 +0x6d fp=0x6ff08 sp=0x6ff00
runtime.mstart()
  c:/go/src/runtime/proc.c:818 fp=0x6ff10 sp=0x6ff08

goroutine 1 [garbage collection, locked to thread]:
runtime.switchtoM()
  c:/go/src/runtime/asm_amd64.s:198 fp=0xc0820676d8 sp=0xc0820676d0
runtime.gogc(0x0)
  c:/go/src/runtime/malloc.go:469 +0x1d6 fp=0xc082067710 sp=0xc0820676d8
runtime.mallocgc(0xa0, 0x9f0bc0, 0xc000000000, 0x220000)
  c:/go/src/runtime/malloc.go:341 +0x398 fp=0xc0820677c0 sp=0xc082067710
runtime.newarray(0x9f0bc0, 0x4, 0x60000c081fffb4f)
  c:/go/src/runtime/malloc.go:365 +0xc8 fp=0xc0820677f8 sp=0xc0820677c0
runtime.growslice(0x8c8700, 0xc08200c3c0, 0x2, 0x2, 0x1, 0x0, 0x0, 0x0)
  c:/go/src/runtime/slice.go:87 +0x2c2 fp=0xc082067858 sp=0xc0820677f8
regexp/syntax.(*compiler).rune(0xc082044078, 0xc08200e950, 0x4, 0x4, 0xc0000000d4, 0x0)
  c:/go/src/regexp/syntax/compile.go:267 +0x102 fp=0xc082067920 sp=0xc082067858
regexp/syntax.(*compiler).compile(0xc082044078, 0xc082002310, 0x0)
  c:/go/src/regexp/syntax/compile.go:119 +0x2e6 fp=0xc082067a00 sp=0xc082067920
regexp/syntax.(*compiler).compile(0xc082044078, 0xc082002380, 0x0)
  c:/go/src/regexp/syntax/compile.go:142 +0x6be fp=0xc082067ae0 sp=0xc082067a00
regexp/syntax.(*compiler).compile(0xc082044078, 0xc082002460, 0x0)
  c:/go/src/regexp/syntax/compile.go:156 +0x9a4 fp=0xc082067bc0 sp=0xc082067ae0
regexp/syntax.Compile(0xc082002460, 0xc082002460, 0x0, 0x0)
  c:/go/src/regexp/syntax/compile.go:83 +0x7f fp=0xc082067c88 sp=0xc082067bc0
regexp.compile(0xa96a90, 0x9, 0xc0820000d4, 0xc08206fb60, 0x0, 0x0)
  c:/go/src/regexp/regexp.go:161 +0x119 fp=0xc082067d30 sp=0xc082067c88
regexp.Compile(0xa96a90, 0x9, 0x1, 0x0, 0x0)
  c:/go/src/regexp/regexp.go:118 +0x57 fp=0xc082067d68 sp=0xc082067d30
regexp.MustCompile(0xa96a90, 0x9, 0x22dc40)
  c:/go/src/regexp/regexp.go:219 +0x47 fp=0xc082067e00 sp=0xc082067d68
github.com/tolsen/slogger/v2.init()
  c:/Users/Administrator/Documents/GitHub/mms-automation/go_planner/src/github.com/tolsen/slogger/v2/context.go:59 +0x93 fp=0xc082067e20 sp=0xc082067e00
github.com/tolsen/slogger/v2/rolling_file_appender.init()
  c:/Users/Administrator/Documents/GitHub/mms-automation/go_planner/src/github.com/tolsen/slogger/v2/rolling_file_appender/rolling_file_appender.go:460 +0x6a fp=0xc082067e40 sp=0xc082067e20
com.tengen/cm/util.init()
  c:/Users/Administrator/Documents/GitHub/mms-automation/go_planner/src/com.tengen/cm/util/uri.go:9 +0x89 fp=0xc082067f30 sp=0xc082067e40
com.tengen/cm/auth/authtest.init()
  c:/Users/Administrator/Documents/GitHub/mms-automation/go_planner/src/com.tengen/cm/auth/authtest/testharness.go:261 +0x53 fp=0xc082067f38 sp=0xc082067f30
com.tengen/cm/action.init()
  c:/Users/Administrator/Documents/GitHub/mms-automation/go_planner/src/com.tengen/cm/action/updateshards_test.go:195 +0x5b fp=0xc082067f80 sp=0xc082067f38
main.init()
  com.tengen/cm/action/_test/_testmain.go:54 +0x51 fp=0xc082067f98 sp=0xc082067f80
runtime.main()
  c:/go/src/runtime/proc.go:58 +0xeb fp=0xc082067fe0 sp=0xc082067f98
runtime.goexit()
  c:/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc082067fe8 sp=0xc082067fe0

goroutine 2 [runnable]:
runtime.forcegchelper()
  c:/go/src/runtime/proc.go:90 fp=0xc08201ffe0 sp=0xc08201ffd8
runtime.goexit()
  c:/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc08201ffe8 sp=0xc08201ffe0
created by runtime.init�~T��~U~V4
  c:/go/src/runtime/proc.go:87 +0x2c

goroutine 3 [runnable]:
runtime.bgsweep()
  c:/go/src/runtime/mgc0.go:82 fp=0xc08201dfe0 sp=0xc08201dfd8
runtime.goexit()
  c:/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc08201dfe8 sp=0xc08201dfe0
created by gc
  c:/go/src/runtime/mgc0.c:1386

goroutine 4 [runnable]:
runtime.runfinq()
  c:/go/src/runtime/malloc.go:712 fp=0xc08204dfe0 sp=0xc08204dfd8
runtime.goexit()
  c:/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc08204dfe8 sp=0xc08204dfe0
created by runtime.createfing
  c:/go/src/runtime/malloc.go:707 +0x65
exit status 2
FAIL  com.tengen/cm/action  0.035s

The error points to a regexp in a 3rd-party dep, but AFAICT that's a red-herring. I'm able to reproduce the same error with a small number of files found here: https://github.com/leothekim/Invalid-Heap-Pointer-Go-1.4.2-Windows-amd64. The same code does not error on Mac OS X. Setting GODEBUG=invalidptr=0 gets rid of the error on Windows.

Curiously, none of the files depends on each other - they just live in the same package. Even more curiously, the error disappears if I removing one of them and run go test or the built executable. This suggests something to do with importing the 3rd-party deps impacting how init() methods are evaluating, likely with allocating more memory for somethingorotherhandwave.

This also seems related to a number of issues I've seen here with a similar error, though I'm a complete golang n00b, and even less of a Windows person, so I'm not quite sure how to proceed. Still, I hope providing some code that reproduces this is helpful in getting to the bottom of this.

[sbinet]

probably related to issue #9125.
also probably interesting: https://groups.google.com/forum/#!topic/golang-nuts/539iJLHmZdU

[leothekim]

Interesting. Also, I forgot to add that this doesn't reproduce with Go 1.3.3 in the same Windows environment.

[FrankReh]

Since GC found a value of 0x9 in a field it thinks should be a pointer, it
is a similar symptom to the issue we ran into when using C pointer fields
for non pointer values. But I thought I read that the gc in 1.3 was
already strict enough to flag those cases as violations.

The initial goroutine stack size seems to have decreased between 1.3.3 and
1.4.2. runtime/stack.h:StackMin is 8k in 1.3.3 and now is 2k in 1.4.2.
Maybe that is causing another problem to surface. We ran into another
problem where addresses passed to C were not allocated from the heap but
there the value gc was claiming was invalid really had been a valid address
in the past, so not something that looked like 0x9.

On Fri, Feb 27, 2015 at 11:33 AM, Leo Kim notifications@github.com wrote:

Interesting. Also, I forgot to add that this doesn't reproduce with Go
1.3.3 in the same Windows environment.

—
Reply to this email directly or view it on GitHub
#10026 (comment).

[leothekim]

Is there a way to increase stack size easily (e.g. runtime flag, compile-time setting)? Just curious if @FrankReh's intuition is correct, if it's worth it.

[minux]

I don't understand how the size of the default goroutine stack could have affect this. Passing a pointer to C will make that escape, so it shouldn't live on stack, unless you pass a pointer as uintptr to C code, which is already broken code.

[leothekim]

Digging a little further, I think this error comes from one of the 3rd party libraries being initted - gowin32. The use of this function is the likely culprit:
https://github.com/winlabs/gowin32/blob/master/wrappers/winuser.go#L23-25

I don't know what MAKEINTRESOURCE is trying to do, and I'm not entirely clear why this doesn't show up with Go 1.3.3. I'll ping the gowin32 maintainers about this.

[minux]

That MAKEINTRESOURCE function is definitely not OK on Go 1.4+. Essentially, it stores a uint16 value as a pointer to uint16, which mostly probably is an invalid pointer to the GC.

[leothekim]

My understanding from talking to someone working on gowin32: There's an analog macro in the Win32 API, MAKEINTRESOURCE, which casts the integer to a LPCTSTR, which is a pointer to a null terminated string. It's used to simulate function overloading in C. It's apparently not treated like a valid pointer in Windows C/C++ land and thus not subject to garbage collection, which is why they're able to get away with it. The gowin32 library is basically trying to emulate similar functionality.

gowin32 will have to break how its MAKEINTRESOURCE function works, which I suppose is just the reality we're in now.

[alexbrainman]

@leothekim your understanding is correct. gowin32 authors have to change their code if they wont it to work with current (ad future) versions of Go.