Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "found bad pointer in Go heap" on linux-mips64le-mengzhuo builder #35541

Open
bcmills opened this issue Nov 12, 2019 · 5 comments
Milestone

Comments

@mengzhuo

This comment has been minimized.

Copy link
Contributor

@mengzhuo mengzhuo commented Nov 13, 2019

I can't reproduce by all.bash nor dist test ( tried serval times)
Bad memory block?

@cherrymui

This comment has been minimized.

Copy link
Contributor

@cherrymui cherrymui commented Nov 20, 2019

Eyeballing the hex dump, it looks to me that the bad pointer is found in a g struct, in g._defer field.

  • 2019-11-11T16:02:42-f511467/linux-mips64le-mengzhuo:
object=0xc000106300 s.base()=0xc000106000 s.limit=0xc000107f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00052a000		g.stack.lo
 *(object+8) = 0xc000532000		g.stack.hi
 *(object+16) = 0xc00052a380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc0005313e0 <==	g._defer
 *(object+48) = 0xc000080380		g.m
 *(object+56) = 0xc0005310b8		g.sched.sp
 *(object+64) = 0x87038			g.sched.pc
 *(object+72) = 0xc000106300		g.sched.g
 ...

The stack bounds matches the stack of goroutine 67, as well as g.sched.pc and g.sched.sp.

What's interesting is that the bad pointer 0xc0005313e0 is also within the stack bound, in particular, within this frame

io/ioutil.readAll(0x3fdc60, 0xc000203a40, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
	/tmp/workdir-host-linux-mipsle-mengzhuo/go/src/io/ioutil/ioutil.go:36 +0xd8 fp=0xc000531360 sp=0xc0005312e8 pc=0x170790

Presumably it is a stack-allocated defer. I don't know why spanOf does not find the span of the stack, returns a different span instead...

  • 2019-11-12T05:18:25-194ae32/linux-mips64le-mengzhuo

This also looks like the g._defer field in a g. The stack bounds match goroutine 1.

What's more interesting is that findObject and badPointer were complaining about pointer 0xc000346ef8, but in the object dump the field is actually containing 0xc00034a688, a different value.

runtime: pointer 0xc000346ef8 to unallocated span span.base()=0xc000346000 span.limit=0xc000355e00 span.state=0
runtime: found in object at *(0xc000000180+0x28)
object=0xc000000180 s.base()=0xc000000000 s.limit=0xc000001f80 s.spanclass=42 s.elemsize=384 s.state=mSpanInUse
 *(object+0) = 0xc00033c000		g.stack.lo
 *(object+8) = 0xc00034c000		g.stack.hi
 *(object+16) = 0xc00033c380		g.stackguard0 = g.stack.lo + _Stackguard = g.stack.lo + 896
 *(object+24) = 0xffffffffffffffff	g.stackguard1
 *(object+32) = 0x0			g._panic
 *(object+40) = 0xc00034a688 <==	g._defer

I think g._defer can only change synchronously, right? But the goroutine is not running...

Also, the span's address range, [0xc000346000, 0xc000355e00], overlaps with the stack bounds [0xc00033c000, 0xc00034c000], assuming my interpretation of the hex dump is right. How could this be possible?

@aclements any idea how g._defer could change this way, or how span tracking could go weirdly?

@cherrymui

This comment has been minimized.

Copy link
Contributor

@cherrymui cherrymui commented Nov 20, 2019

Another one today https://build.golang.org/log/a810adac990353a9918a366fc13538c03649d206
Also looks like g._defer pointing to the stack.

@mengzhuo

This comment has been minimized.

Copy link
Contributor

@mengzhuo mengzhuo commented Nov 22, 2019

@cherrymui
The PAGESIZE setting of this builder is 16384 bytes and
0xc000346000 % 16384 = 8192

It looks like some bug of mpagealloc?

@mknyszek

This comment has been minimized.

Copy link
Contributor

@mknyszek mknyszek commented Nov 22, 2019

@mengzhuo Only the scavenger (mgcscavenge.go) operates in terms of physical pages, the runtime page size is always 8 KiB on all platforms, so 8 KiB alignment is exactly what we want.

It could be related to the page allocator, but the physical page size here doesn't matter, I don't think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.