runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

bcmills · 2019-11-12T17:27:27Z

2019-11-12T05:18:25-194ae32/linux-mips64le-mengzhuo
2019-11-11T16:02:42-f511467/linux-mips64le-mengzhuo

Possibly related to #34835, #32324.

CC @mengzhuo @cherrymui

mengzhuo · 2019-11-13T15:17:47Z

I can't reproduce by all.bash nor dist test ( tried serval times)
Bad memory block?

cherrymui · 2019-11-20T01:17:22Z

Eyeballing the hex dump, it looks to me that the bad pointer is found in a g struct, in g._defer field.

2019-11-11T16:02:42-f511467/linux-mips64le-mengzhuo:

object=0xc000106300 s.base()=0xc000106000 s.limit=0xc000107f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00052a000		g.stack.lo
 *(object+8) = 0xc000532000		g.stack.hi
 *(object+16) = 0xc00052a380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc0005313e0 <==	g._defer
 *(object+48) = 0xc000080380		g.m
 *(object+56) = 0xc0005310b8		g.sched.sp
 *(object+64) = 0x87038			g.sched.pc
 *(object+72) = 0xc000106300		g.sched.g
 ...

The stack bounds matches the stack of goroutine 67, as well as g.sched.pc and g.sched.sp.

What's interesting is that the bad pointer 0xc0005313e0 is also within the stack bound, in particular, within this frame

io/ioutil.readAll(0x3fdc60, 0xc000203a40, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/io/ioutil/ioutil.go:36 +0xd8 fp=0xc000531360 sp=0xc0005312e8 pc=0x170790

Presumably it is a stack-allocated defer. I don't know why spanOf does not find the span of the stack, returns a different span instead...

2019-11-12T05:18:25-194ae32/linux-mips64le-mengzhuo

This also looks like the g._defer field in a g. The stack bounds match goroutine 1.

What's more interesting is that findObject and badPointer were complaining about pointer 0xc000346ef8, but in the object dump the field is actually containing 0xc00034a688, a different value.

runtime: pointer 0xc000346ef8 to unallocated span span.base()=0xc000346000 span.limit=0xc000355e00 span.state=0
runtime: found in object at *(0xc000000180+0x28)
object=0xc000000180 s.base()=0xc000000000 s.limit=0xc000001f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00033c000		g.stack.lo
 *(object+8) = 0xc00034c000		g.stack.hi
 *(object+16) = 0xc00033c380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc00034a688 <==	g._defer

I think g._defer can only change synchronously, right? But the goroutine is not running...

Also, the span's address range, [0xc000346000, 0xc000355e00], overlaps with the stack bounds [0xc00033c000, 0xc00034c000], assuming my interpretation of the hex dump is right. How could this be possible?

@aclements any idea how g._defer could change this way, or how span tracking could go weirdly?

cherrymui · 2019-11-20T01:22:05Z

Another one today https://build.golang.org/log/a810adac990353a9918a366fc13538c03649d206
Also looks like g._defer pointing to the stack.

mengzhuo · 2019-11-22T14:51:23Z

@cherrymui
The PAGESIZE setting of this builder is 16384 bytes and
0xc000346000 % 16384 = 8192

It looks like some bug of mpagealloc?

mknyszek · 2019-11-22T17:41:21Z

@mengzhuo Only the scavenger (mgcscavenge.go) operates in terms of physical pages, the runtime page size is always 8 KiB on all platforms, so 8 KiB alignment is exactly what we want.

It could be related to the page allocator, but the physical page size here doesn't matter, I don't think.

ianlancetaylor · 2019-12-06T02:55:48Z

2019-12-05T17:23:50-50535e6/linux-mips64le-mengzhuo
2019-12-05T01:48:12-fa3a121/linux-mips64le-mengzhuo
2019-12-03T15:26:50-2ac1ca9/linux-mips64le-mengzhuo
2019-11-19T19:59:21-8b1e8a4/linux-mips64le-mengzhuo
2019-11-18T22:08:03-fff4502/linux-mips64le-mengzhuo
2019-11-15T20:56:35-9af8794/freebsd-arm64-dmgk
2019-11-15T02:31:58-498eaee/linux-mips64le-mengzhuo
2019-11-14T00:58:33-e77106c/linux-mips64le-mengzhuo
2019-11-12T22:28:47-13fb147/linux-mips64le-mengzhuo

bcmills · 2020-01-22T14:32:52Z

Still occurring regularly. Does this need to be addressed for the 1.14 release? Should we mention it as a known issue in the release notes?
(CC @cagedmantis @dmitshur @toothrot)

2020-01-21T23:58:54-fb8cdb7/linux-mips64le-mengzhuo
2020-01-07T18:40:18-7340d15/linux-mips64le-mengzhuo
2019-12-20T23:07:52-4d5bb9c/linux-mips64le-mengzhuo
2019-12-18T19:09:23-87a5467/linux-mips64le-mengzhuo
2019-12-10T21:11:20-207a0b7/linux-mips64le-mengzhuo

toothrot · 2020-01-22T15:50:00Z

I think we would need input from the runtime team to make that decision, or anyone else on this thread with sufficient expertise (@cherrymui @ianlancetaylor @aclements @rsc @randall77)

It's interesting that the failure also happened on freebsd-arm64, as @ianlancetaylor linked. Should the title be updated?

bcmills · 2020-01-22T15:52:24Z

I'm not sure whether the freebsd-arm64-dmgk failure is the same underlying cause.

(I'd prefer to keep this issue focused on the mips64le builder unless we have a specific reason to suspect the same root cause for both.)

cherrymui · 2020-01-22T15:57:56Z

I looked at a few new MIPS failures and they all look like the ones before, and are consistent with my previous comment. The "bad pointer" seems from g._defer, which points to the stack (within the stack bounds), but spanOf returns a different span. I don't yet understand how this could happen.

The freebsd-arm64 one is different, and may be fixed by CL https://go-review.googlesource.com/c/go/+/212626 .

mengzhuo · 2020-01-22T15:59:37Z

I think it’s unnecessary. The kernel development of Loongson is out of tree and not stable. I’ve tried to use Linux 5.4, but it has an unknown bug in vsyscall and causing segment fault. Bryan C. Mills <notifications@github.com>于2020年1月22日周三22:32写道：

…

Still occurring regularly. Does this need to be addressed for the 1.14 release? Should we mention it as a known issue in the release notes? (CC @cagedmantis <https://github.com/cagedmantis> @dmitshur <https://github.com/dmitshur> @toothrot <https://github.com/toothrot>) 2020-01-21T23:58:54-fb8cdb7/linux-mips64le-mengzhuo <https://build.golang.org/log/fa85cfe17b4648348e5b5de9a069f2afceabaeb1> 2020-01-07T18:40:18-7340d15/linux-mips64le-mengzhuo <https://build.golang.org/log/6f063f52e50c29c5afb522fa6d8a0f340cfd7418> 2019-12-20T23:07:52-4d5bb9c/linux-mips64le-mengzhuo <https://build.golang.org/log/7592f9e2accda55b972da33ac348f133bf6fdc69> 2019-12-18T19:09:23-87a5467/linux-mips64le-mengzhuo <https://build.golang.org/log/26562517daa862c4feabd0c1daa0ea3f6e6674f1> 2019-12-10T21:11:20-207a0b7/linux-mips64le-mengzhuo <https://build.golang.org/log/2ee2b502cf3b7af0c3f495e46655eb59e82d05b4> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35541?email_source=notifications&email_token=AAGYHHWHUFDNI66XBRV3QGTQ7BKJTA5CNFSM4JMGUVW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJTYB2Y#issuecomment-577208555>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGYHHS2HWBNR7W726HEPLDQ7BKJTANCNFSM4JMGUVWQ> .

bcmills · 2020-02-20T14:03:50Z

2020-02-20T03:27:46-715e8bb/linux-mips64le-mengzhuo
2020-02-19T21:00:02-1bf9d3c/linux-mips64le-mengzhuo
2020-02-17T10:30:03-88e564e/linux-mips64le-mengzhuo
2020-02-12T17:32:01-7385947/linux-mips64le-mengzhuo
2020-01-31T00:36:45-25d0ffe/linux-mips64le-mengzhuo
2020-01-29T02:43:46-a6701d8/linux-mips64le-mengzhuo
2020-01-24T19:37:58-67539f6/linux-mips64le-mengzhuo

mengzhuo · 2020-02-20T14:20:20Z

@bcmills It weird that mips-rtrk don't crash not even once by this "unused span".
As I said before the kernel development of Loogson has a lots of patches and out of tree for a long time.

Here is kernel that loongson used.
http://dev.lemote.com:8000/cgit/linux-official.git/log/?h=linux-4.19

bcmills · 2020-03-05T14:36:12Z

This strongly correlates to go/types.(*Checker) (#37602). All of these failures have it on the stack:

CC @griesemer

cherrymui · 2020-03-05T15:50:46Z

This strongly correlates to go/types.(*Checker)

I noticed that a while ago and looked at that function, but didn't find anything suspicious, especially about defers.

mengzhuo · 2020-03-19T03:47:37Z

Hi @cherrymui

I've tried turn on debugCachedWork at runtime/mgc.go, now it failed the test every time.

runtime: pointer 0xc0002a9068 to unused region of span span.base()=0xc00016c000 span.limit=0xc0001b9808 span.state=1
runtime: found in object at *(0xc0002ed880+0x28)                                                                    
object=0xc0002ed880 s.base()=0xc0002ec000 s.limit=0xc0002edff0 s.spanclass=16 s.elemsize=112 s.state=mSpanInUse     
 *(object+0) = 0x1000000000020                                                                                      
 *(object+8) = 0xc0002a8dd8                                                                                         
 *(object+16) = 0x2aba30                                                                                            
 *(object+24) = 0x0                                                                                                 
 *(object+32) = 0x0                                                                                                 
 *(object+40) = 0xc0002a9068 <==                                                                                    
 *(object+48) = 0x733f20                                                                                            
 *(object+56) = 0xc0002a8fb0                                                                                        
 *(object+64) = 0x2ab24c                                                                                            
 *(object+72) = 0x0                                                                                                 
 *(object+80) = 0x0                                                                                                 
 *(object+88) = 0x0                                                                                                 
 *(object+96) = 0x0                                                                                                 
 *(object+104) = 0x0                                                                                                
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)                                         
                                                                                                                    
runtime stack:                                                                                                      
runtime.throw(0x72efec, 0x3e)                                                                                       
        /root/godev/src/runtime/panic.go:1114 +0x6c fp=0xc00003be30 sp=0xc00003be08 pc=0x4ef94                      
runtime.badPointer(0xffcded4338, 0xc0002a9068, 0xc0002ed880, 0x28)                                                  
        /root/godev/src/runtime/mbitmap.go:380 +0x2c4 fp=0xc00003be70 sp=0xc00003be30 pc=0x27cac                    
runtime.findObject(0xc0002a9068, 0xc0002ed880, 0x28, 0x0, 0x0, 0x0)                                                 
        /root/godev/src/runtime/mbitmap.go:416 +0xb8 fp=0xc00003bea0 sp=0xc00003be70 pc=0x27d68                     
runtime.scanobject(0xc0002ed880, 0xc000021e98)                                                                      
        /root/godev/src/runtime/mgcmark.go:1274 +0x2a8 fp=0xc00003bf38 sp=0xc00003bea0 pc=0x36060                   
runtime.gcDrain(0xc000021e98, 0x3)                                                                                  
        /root/godev/src/runtime/mgcmark.go:1032 +0x2b0 fp=0xc00003bf98 sp=0xc00003bf38 pc=0x35668                   
runtime.gcBgMarkWorker.func2()                                                                                      
        /root/godev/src/runtime/mgc.go:1932 +0x88 fp=0xc00003bfd0 sp=0xc00003bf98 pc=0x85d20                        
runtime.systemstack(0x4d2)                                                                                          
        /root/godev/src/runtime/asm_mips64x.s:206 +0xa8 fp=0xc00003bfd8 sp=0xc00003bfd0 pc=0x893d8                  
runtime.mstart()                                                                                                    
        /root/godev/src/runtime/proc.go:1042 fp=0xc00003bfd8 sp=0xc00003bfd8 pc=0x55348

mengzhuo · 2020-04-26T09:15:21Z

Hi, @cherrymui

I can reproduce this bug on my box by running runtime/TestDeferHeapAndStack test.
The builder trybot using short test mode which won't trigger this bug.

=== RUN   TestDeferHeapAndStack
runtime: pointer 0xc0004a9ae8 to unallocated span span.base()=0xc0004a8000 span.limit=0xc0004aa000 span.state=0
runtime: found in object at *(0xc00028bc20+0x28)
object=0xc00028bc20 s.base()=0xc00028a000 s.limit=0xc00028bfe0 s.spanclass=12 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0x10000000008
 *(object+8) = 0xc0004a99f0
 *(object+16) = 0x12021178c
 *(object+24) = 0x1202d9340
 *(object+32) = 0x0
 *(object+40) = 0xc0004a9ae8 <==
 *(object+48) = 0x0
 *(object+56) = 0x0
 *(object+64) = 0x0
 *(object+72) = 0xc000687840
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x1202d4b7a, 0x3e)
	/root/godev/src/runtime/panic.go:1116 +0x6c fp=0xffca57e720 sp=0xffca57e6f8 pc=0x120041e2c
runtime.badPointer(0xffc9545718, 0xc0004a9ae8, 0xc00028bc20, 0x28)
	/root/godev/src/runtime/mbitmap.go:380 +0x2c4 fp=0xffca57e760 sp=0xffca57e720 pc=0x12001a644
runtime.findObject(0xc0004a9ae8, 0xc00028bc20, 0x28, 0x0, 0x0, 0x0)
	/root/godev/src/runtime/mbitmap.go:416 +0xb8 fp=0xffca57e790 sp=0xffca57e760 pc=0x12001a700
runtime.scanobject(0xc00028bc20, 0xc000026698)
	/root/godev/src/runtime/mgcmark.go:1383 +0x2f0 fp=0xffca57e828 sp=0xffca57e790 pc=0x120029508
runtime.gcDrainN(0xc000026698, 0x10000, 0x12007e30c)
	/root/godev/src/runtime/mgcmark.go:1235 +0x1a0 fp=0xffca57e850 sp=0xffca57e828 pc=0x120028f08
runtime.gcAssistAlloc1(0xc000001c80, 0x10000)
	/root/godev/src/runtime/mgcmark.go:636 +0x140 fp=0xffca57e898 sp=0xffca57e850 pc=0x120027490
runtime.gcAssistAlloc.func1()
	/root/godev/src/runtime/mgcmark.go:547 +0x44 fp=0xffca57e8b0 sp=0xffca57e898 pc=0x12007ccec
runtime.systemstack(0x0)
	/root/godev/src/runtime/asm_mips64x.s:206 +0xa8 fp=0xffca57e8b8 sp=0xffca57e8b0 pc=0x120084b08
runtime.mstart()
	/root/godev/src/runtime/proc.go:1094 fp=0xffca57e8b8 sp=0xffca57e8b8 pc=0x120048460

goroutine 8 [GC assist marking]:
runtime.systemstack_switch()
	/root/godev/src/runtime/asm_mips64x.s:158 +0x8 fp=0xc0002f7060 sp=0xc0002f7058 pc=0x120084a40
runtime.gcAssistAlloc(0xc000001c80)
	/root/godev/src/runtime/mgcmark.go:546 +0x1b8 fp=0xc0002f70b8 sp=0xc0002f7060 pc=0x120027180
runtime.mallocgc(0x8, 0x1202608e0, 0xffca582701, 0xc0000e8400)
	/root/godev/src/runtime/malloc.go:950 +0x8dc fp=0xc0002f7178 sp=0xc0002f70b8 pc=0x1200110cc
runtime.newobject(0x1202608e0, 0xc0000e8000)
	/root/godev/src/runtime/malloc.go:1187 +0x4c fp=0xc0002f71a0 sp=0xc0002f7178 pc=0x1200113f4
runtime_test.deferHeapAndStack(0x69, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:827 +0x4c fp=0xc0002f7270 sp=0xc0002f71a0 pc=0x12021171c
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 1 [chan receive, locked to thread]:
runtime.gopark(0x1202d7350, 0xc000064298, 0x1204a170e, 0x2)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000125aa8 sp=0xc000125a90 pc=0x120045398
runtime.chanrecv(0xc000064240, 0xc000125bb0, 0xc000000101, 0x1201100a0)
	/root/godev/src/runtime/chan.go:567 +0x398 fp=0xc000125b30 sp=0xc000125aa8 pc=0x120008500
runtime.chanrecv1(0xc000064240, 0xc000125bb0)
	/root/godev/src/runtime/chan.go:434 +0x28 fp=0xc000125b58 sp=0xc000125b30 pc=0x120008118
testing.(*T).Run(0xc00011e280, 0x1202c88ec, 0x15, 0x1202d86a0, 0x12010fc01)
	/root/godev/src/testing/testing.go:1074 +0x398 fp=0xc000125c00 sp=0xc000125b58 pc=0x1201100c0
testing.runTests.func1(0xc00011e140)
	/root/godev/src/testing/testing.go:1342 +0x8c fp=0xc000125c48 sp=0xc000125c00 pc=0x120114a9c
testing.tRunner(0xc00011e140, 0xc000125d40)
	/root/godev/src/testing/testing.go:1022 +0xf8 fp=0xc000125c90 sp=0xc000125c48 pc=0x12010fca0
testing.runTests(0xc00000e100, 0x1204ab540, 0x153, 0x153, 0xbfa16e2ad25438c0, 0x8bb2d601f1, 0x1204c75e0, 0xfff2a1d978)
	/root/godev/src/testing/testing.go:1340 +0x328 fp=0xc000125d60 sp=0xc000125c90 pc=0x120111878
testing.(*M).Run(0xc00011a000, 0x0)
	/root/godev/src/testing/testing.go:1250 +0x1e0 fp=0xc000125e88 sp=0xc000125d60 pc=0x120110788
runtime_test.TestMain(0xc00011a000)
	/root/godev/src/runtime/crash_test.go:28 +0x3c fp=0xc000125ed0 sp=0xc000125e88 pc=0x1201ae844
main.main()
	_testmain.go:1209 +0x198 fp=0xc000125f80 sp=0xc000125ed0 pc=0x12023c050
runtime.main()
	/root/godev/src/runtime/proc.go:204 +0x2cc fp=0xc000125fd8 sp=0xc000125f80 pc=0x120044e54
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc000125fd8 sp=0xc000125fd8 pc=0x120086ed4

goroutine 2 [force gc (idle)]:
runtime.gopark(0x1202d7660, 0x1204c6d40, 0x1411, 0x1)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002efb0 sp=0xc00002ef98 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.forcegchelper()
	/root/godev/src/runtime/proc.go:255 +0x108 fp=0xc00002efd8 sp=0xc00002efb0 pc=0x1200451f8
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002efd8 sp=0xc00002efd8 pc=0x120086ed4
created by runtime.init.5
	/root/godev/src/runtime/proc.go:243 +0x48

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1202d7660, 0x1204c7160, 0x140c, 0x1)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002f7a8 sp=0xc00002f790 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.bgsweep(0xc000056000)
	/root/godev/src/runtime/mgcsweep.go:90 +0x200 fp=0xc00002f7d0 sp=0xc00002f7a8 pc=0x12002d1d8
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002f7d0 sp=0xc00002f7d0 pc=0x120086ed4
created by runtime.gcenable
	/root/godev/src/runtime/mgc.go:217 +0x74

goroutine 4 [sleep]:
runtime.gopark(0x1202d7660, 0x1204c7120, 0x1313, 0x2)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002af28 sp=0xc00002af10 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.scavengeSleep(0xebdfb, 0x12b3d0)
	/root/godev/src/runtime/mgcscavenge.go:214 +0x100 fp=0xc00002af78 sp=0xc00002af28 pc=0x12002b038
runtime.bgscavenge(0xc000056000)
	/root/godev/src/runtime/mgcscavenge.go:316 +0x1e8 fp=0xc00002afd0 sp=0xc00002af78 pc=0x12002b248
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002afd0 sp=0xc00002afd0 pc=0x120086ed4
created by runtime.gcenable
	/root/godev/src/runtime/mgc.go:218 +0x9c

goroutine 5 [finalizer wait]:
runtime.gopark(0x1202d7660, 0x1204ef6a0, 0x1410, 0x1)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000030750 sp=0xc000030738 pc=0x120045398
runtime.goparkunlock(...)
	/root/godev/src/runtime/proc.go:312
runtime.runfinq()
	/root/godev/src/runtime/mfinal.go:175 +0xf0 fp=0xc0000307d8 sp=0xc000030750 pc=0x12001fc78
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc0000307d8 sp=0xc0000307d8 pc=0x120086ed4
created by runtime.createfing
	/root/godev/src/runtime/mfinal.go:156 +0xb0

goroutine 6 [chan receive]:
runtime.gopark(0x1202d7350, 0xc0000642f8, 0xfff2a1170e, 0x2)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002e670 sp=0xc00002e658 pc=0x120045398
runtime.chanrecv(0xc0000642a0, 0x0, 0xc000001901, 0x120211604)
	/root/godev/src/runtime/chan.go:567 +0x398 fp=0xc00002e6f8 sp=0xc00002e670 pc=0x120008500
runtime.chanrecv1(0xc0000642a0, 0x0)
	/root/godev/src/runtime/chan.go:434 +0x28 fp=0xc00002e720 sp=0xc00002e6f8 pc=0x120008118
runtime_test.TestDeferHeapAndStack(0xc00011e280)
	/root/godev/src/runtime/stack_test.go:822 +0x1b0 fp=0xc00002e780 sp=0xc00002e720 pc=0x120211640
testing.tRunner(0xc00011e280, 0x1202d86a0)
	/root/godev/src/testing/testing.go:1022 +0xf8 fp=0xc00002e7c8 sp=0xc00002e780 pc=0x12010fca0
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002e7c8 sp=0xc00002e7c8 pc=0x120086ed4
created by testing.(*T).Run
	/root/godev/src/testing/testing.go:1073 +0x378

goroutine 7 [runnable]:
runtime.gcStart(0x0, 0x0, 0xc000000000)
	/root/godev/src/runtime/mgc.go:1232 +0x1c fp=0xc0001e32f8 sp=0xc0001e32f8 pc=0x12002290c
runtime.mallocgc(0x400, 0x120261f20, 0xc000490401, 0xc0005003d8)
	/root/godev/src/runtime/malloc.go:1151 +0x454 fp=0xc0001e33b8 sp=0xc0001e32f8 pc=0x120010c44
runtime.newobject(0x120261f20, 0x38)
	/root/godev/src/runtime/malloc.go:1187 +0x4c fp=0xc0001e33e0 sp=0xc0001e33b8 pc=0x1200113f4
runtime_test.deferHeapAndStack(0x1d, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:848 +0x128 fp=0xc0001e34b0 sp=0xc0001e33e0 pc=0x1202117f8
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 9 [runnable]:
runtime_test.deferHeapAndStack(0x39, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:827 +0x20 fp=0xc0005a0b70 sp=0xc0005a0b70 pc=0x1202116f0
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 10 [runnable]:
runtime_test.deferHeapAndStack(0x43, 0x1202d9340)
	/root/godev/src/runtime/stack_test.go:827 +0x20 fp=0xc0004a3390 sp=0xc0004a3390 pc=0x1202116f0
...additional frames elided...
created by runtime_test.TestDeferHeapAndStack
	/root/godev/src/runtime/stack_test.go:812 +0x174

goroutine 11 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000413600, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00002a760 sp=0xc00002a748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000020000)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc00002a7d0 sp=0xc00002a760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00002a7d0 sp=0xc00002a7d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000500010, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000092760 sp=0xc000092748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000022800)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc0000927d0 sp=0xc000092760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc0000927d0 sp=0xc0000927d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000500020, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc000031760 sp=0xc000031748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000025000)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc0000317d0 sp=0xc000031760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc0000317d0 sp=0xc0000317d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x1202d7460, 0xc000500030, 0x1418, 0x0)
	/root/godev/src/runtime/proc.go:306 +0x110 fp=0xc00008e760 sp=0xc00008e748 pc=0x120045398
runtime.gcBgMarkWorker(0xc000027800)
	/root/godev/src/runtime/mgc.go:1895 +0x11c fp=0xc00008e7d0 sp=0xc00008e760 pc=0x1200247a4
runtime.goexit()
	/root/godev/src/runtime/asm_mips64x.s:646 +0x4 fp=0xc00008e7d0 sp=0xc00008e7d0 pc=0x120086ed4
created by runtime.gcBgMarkStartWorkers
	/root/godev/src/runtime/mgc.go:1843 +0xc0
exit status 2
FAIL	runtime	0.144s

mengzhuo · 2020-05-08T11:38:34Z

I did a git bisect HEAD go1.13.9 and found that this commit is the cause of this issue
a762221

cherrymui · 2020-05-08T14:03:01Z

Thanks @mengzhuo . These are all useful information. I'll take a careful look when I get a chance. Thanks.

gopherbot · 2020-05-18T18:24:20Z

Change https://golang.org/cl/234478 mentions this issue: runtime: add a barrier in copystack before adjusting pointers

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 12, 2019

bcmills added this to the Go1.14 milestone Nov 12, 2019

bcmills mentioned this issue Nov 12, 2019

runtime: "found bad pointer in Go heap" on darwin/arm64 #32324

Closed

toothrot modified the milestones: Go1.14, Go1.15 Feb 25, 2020

This was referenced Feb 27, 2020

runtime: "found bad pointer in Go heap" on solaris-amd64-oraclerel builder #37502

Open

go/types: SIGSEGV in (*Checker).exprInternal #37602

Open

runtime: fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) #37688

Closed

gopherbot closed this as completed in c53b2bd May 21, 2020

mengzhuo mentioned this issue Jan 15, 2021

runtime: SIGSEGV in runtime.deltimer on linux-mips-rtrk during ReadMemStats #43712

Closed

golang locked and limited conversation to collaborators May 21, 2021

gopherbot added the FrozenDueToAge label May 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

bcmills commented Nov 12, 2019

mengzhuo commented Nov 13, 2019

cherrymui commented Nov 20, 2019

cherrymui commented Nov 20, 2019

mengzhuo commented Nov 22, 2019

mknyszek commented Nov 22, 2019

ianlancetaylor commented Dec 6, 2019

bcmills commented Jan 22, 2020

toothrot commented Jan 22, 2020

bcmills commented Jan 22, 2020

cherrymui commented Jan 22, 2020

mengzhuo commented Jan 22, 2020 via email

bcmills commented Feb 20, 2020

mengzhuo commented Feb 20, 2020

bcmills commented Mar 5, 2020 •

edited

Loading

cherrymui commented Mar 5, 2020

mengzhuo commented Mar 19, 2020

mengzhuo commented Apr 26, 2020 •

edited

Loading

mengzhuo commented May 8, 2020

cherrymui commented May 8, 2020

gopherbot commented May 18, 2020

runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

Comments

bcmills commented Nov 12, 2019

mengzhuo commented Nov 13, 2019

cherrymui commented Nov 20, 2019

cherrymui commented Nov 20, 2019

mengzhuo commented Nov 22, 2019

mknyszek commented Nov 22, 2019

ianlancetaylor commented Dec 6, 2019

bcmills commented Jan 22, 2020

toothrot commented Jan 22, 2020

bcmills commented Jan 22, 2020

cherrymui commented Jan 22, 2020

mengzhuo commented Jan 22, 2020 via email

bcmills commented Feb 20, 2020

mengzhuo commented Feb 20, 2020

bcmills commented Mar 5, 2020 • edited Loading

cherrymui commented Mar 5, 2020

mengzhuo commented Mar 19, 2020

mengzhuo commented Apr 26, 2020 • edited Loading

mengzhuo commented May 8, 2020

cherrymui commented May 8, 2020

gopherbot commented May 18, 2020

bcmills commented Mar 5, 2020 •

edited

Loading

mengzhuo commented Apr 26, 2020 •

edited

Loading