Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "found bad pointer in Go heap" on darwin/arm64 #50063

Closed
bcmills opened this issue Dec 9, 2021 · 11 comments
Closed

runtime: "found bad pointer in Go heap" on darwin/arm64 #50063

bcmills opened this issue Dec 9, 2021 · 11 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Dec 9, 2021

runtime: pointer 0x14000270000 to unallocated span span.base()=0x14000266000 span.limit=0x14000271d00 span.state=0
runtime: found in object at *(0x14000130050+0x8)
object=0x14000130050 s.base()=0x14000130000 s.limit=0x14000131fe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0x140006d4257
 *(object+8) = 0x5 <==
 *(object+16) = 0x16f923847
 *(object+24) = 0x7
 *(object+32) = 0x14000026e20
 *(object+40) = 0x1c
 *(object+48) = 0x1400002a07f
 *(object+56) = 0x10
 *(object+64) = 0x1
 *(object+72) = 0x1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x10097fa76?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:992 +0x50 fp=0x16fe96d40 sp=0x16fe96d10 pc=0x10050eda0
runtime.badPointer(0x107e885e0, 0x14000035238?, 0x14000130050, 0x16fe96e10?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:368 +0x120 fp=0x16fe96d90 sp=0x16fe96d40 pc=0x1004efaf0
runtime.findObject(0x14000780f60?, 0x1005158e0?, 0x140001024e0?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:410 +0xb4 fp=0x16fe96dd0 sp=0x16fe96d90 pc=0x1004efc54
runtime.scanobject(0x14000130050, 0x14000035238)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1316 +0x150 fp=0x16fe96e60 sp=0x16fe96dd0 pc=0x1004fa8c0
runtime.gcDrain(0x14000035238, 0x3)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1081 +0x1c8 fp=0x16fe96ec0 sp=0x16fe96e60 pc=0x1004fa0b8
runtime.gcBgMarkWorker.func2()
	/tmp/buildlet/go/src/runtime/mgc.go:1276 +0x98 fp=0x16fe96f10 sp=0x16fe96ec0 pc=0x1004f6f98
runtime.systemstack()
	/tmp/buildlet/go/src/runtime/asm_arm64.s:237 +0x6c fp=0x16fe96f20 sp=0x16fe96f10 pc=0x10053c23c

greplogs --dashboard -md -l -e '(?ms)\Adarwin-arm64.*fatal error: found bad pointer in Go heap'

2021-12-08T17:24:46-ac7e950/darwin-arm64-12_0-toothrot
(Note two-year gap!)
2019-09-26T17:34:54-430d2aa/darwin-arm64-corellium
2019-09-25T16:33:28-8189a06/darwin-arm64-corellium
2019-08-29T19:54:46-75198b9/darwin-arm64-corellium
2019-08-27T21:02:43-cc6feab/darwin-arm64-corellium
2019-07-03T07:14:40-e2fdce9/darwin-arm64-corellium
2019-06-26T20:10:05-fc26cba/darwin-arm64-corellium

@bcmills bcmills added OS-Darwin NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 9, 2021
@bcmills bcmills added this to the Go1.18 milestone Dec 9, 2021
@bcmills
Copy link
Member Author

bcmills commented Dec 9, 2021

Note that darwin/arm64 is a first class port.

CC @jeremyfaller for routing.

@cherrymui
Copy link
Member

cherrymui commented Dec 9, 2021

Before the "gap" the darwin-arm64-corellium was the iOS builder. They may not be related.

@cherrymui cherrymui reopened this Dec 9, 2021
@cherrymui
Copy link
Member

cherrymui commented Dec 9, 2021

Bad click. No intention to close.

@bcmills
Copy link
Member Author

bcmills commented Dec 9, 2021

Before the "gap" the darwin-arm64-corellium was the iOS builder. They may not be related.

Agreed. (I'm much more concerned about this new one, because I think it's more likely to reflect a source of crashes on real users' hardware.)

@cherrymui
Copy link
Member

cherrymui commented Dec 9, 2021

From the message it seems the bad pointer is 0x14000270000, found at *(0x14000130050+0x8), but later on

*(object+8) = 0x5 <==

It looks like the memory content is changing between the two reads, so there is something concurrently writing to the object. But 5 is also not a valid pointer value. So from the other writer's perspective this field should not be a pointer?

@cherrymui
Copy link
Member

cherrymui commented Dec 9, 2021

This may be interesting, or may be completely red-herring:

The object's content looks a lot like some frames' arguments on some of the stacks:

 *(object+0) = 0x140006d4257
 *(object+8) = 0x5 <==
 *(object+16) = 0x16f923847
 *(object+24) = 0x7
 *(object+32) = 0x14000026e20
 *(object+40) = 0x1c
 *(object+48) = 0x1400002a07f
 *(object+56) = 0x10
 *(object+64) = 0x1
 *(object+72) = 0x1

goroutine 262

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4261, 0x3}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 264

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4271, 0x6}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 135

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4151, 0x7}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 266

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4291, 0x7}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

object+16 and later words match exactly with the third and later arguments. object+0 and object+8 are also quite close to the second arguments. However, the object's address, 0x14000130050, doesn't match any of the stacks, and it is on an mSpanInUse span instead of a mSpanManual.

@randall77
Copy link
Contributor

randall77 commented Dec 9, 2021

@cherrymui Could it be a layout object (stackArgs?) used by reflect.Call?

@cherrymui
Copy link
Member

cherrymui commented Dec 9, 2021

That function is not called by reflect call, so probably not.

@randall77
Copy link
Contributor

randall77 commented Dec 9, 2021

Could also be a closure, I guess.

@cherrymui
Copy link
Member

cherrymui commented Dec 9, 2021

That call is at https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/load/pkg.go;l=1028
It is in a closure but the closure would capture parent, instead of parent.ImportPath, parent.Dir, parent.Root and other arguments in that order. The order of the fields in the bad object also doesn't match the parent object.

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jan 29, 2022

This has not happened again. Optimistically closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
None yet
Development

No branches or pull requests

4 participants