Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "found bad pointer in Go heap" on darwin/arm64 #50063

Open
bcmills opened this issue Dec 9, 2021 · 10 comments
Open

runtime: "found bad pointer in Go heap" on darwin/arm64 #50063

bcmills opened this issue Dec 9, 2021 · 10 comments

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Dec 9, 2021

runtime: pointer 0x14000270000 to unallocated span span.base()=0x14000266000 span.limit=0x14000271d00 span.state=0
runtime: found in object at *(0x14000130050+0x8)
object=0x14000130050 s.base()=0x14000130000 s.limit=0x14000131fe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0x140006d4257
 *(object+8) = 0x5 <==
 *(object+16) = 0x16f923847
 *(object+24) = 0x7
 *(object+32) = 0x14000026e20
 *(object+40) = 0x1c
 *(object+48) = 0x1400002a07f
 *(object+56) = 0x10
 *(object+64) = 0x1
 *(object+72) = 0x1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x10097fa76?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:992 +0x50 fp=0x16fe96d40 sp=0x16fe96d10 pc=0x10050eda0
runtime.badPointer(0x107e885e0, 0x14000035238?, 0x14000130050, 0x16fe96e10?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:368 +0x120 fp=0x16fe96d90 sp=0x16fe96d40 pc=0x1004efaf0
runtime.findObject(0x14000780f60?, 0x1005158e0?, 0x140001024e0?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:410 +0xb4 fp=0x16fe96dd0 sp=0x16fe96d90 pc=0x1004efc54
runtime.scanobject(0x14000130050, 0x14000035238)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1316 +0x150 fp=0x16fe96e60 sp=0x16fe96dd0 pc=0x1004fa8c0
runtime.gcDrain(0x14000035238, 0x3)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1081 +0x1c8 fp=0x16fe96ec0 sp=0x16fe96e60 pc=0x1004fa0b8
runtime.gcBgMarkWorker.func2()
	/tmp/buildlet/go/src/runtime/mgc.go:1276 +0x98 fp=0x16fe96f10 sp=0x16fe96ec0 pc=0x1004f6f98
runtime.systemstack()
	/tmp/buildlet/go/src/runtime/asm_arm64.s:237 +0x6c fp=0x16fe96f20 sp=0x16fe96f10 pc=0x10053c23c

greplogs --dashboard -md -l -e '(?ms)\Adarwin-arm64.*fatal error: found bad pointer in Go heap'

2021-12-08T17:24:46-ac7e950/darwin-arm64-12_0-toothrot
(Note two-year gap!)
2019-09-26T17:34:54-430d2aa/darwin-arm64-corellium
2019-09-25T16:33:28-8189a06/darwin-arm64-corellium
2019-08-29T19:54:46-75198b9/darwin-arm64-corellium
2019-08-27T21:02:43-cc6feab/darwin-arm64-corellium
2019-07-03T07:14:40-e2fdce9/darwin-arm64-corellium
2019-06-26T20:10:05-fc26cba/darwin-arm64-corellium

@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 9, 2021

Note that darwin/arm64 is a first class port.

CC @jeremyfaller for routing.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Dec 9, 2021

Before the "gap" the darwin-arm64-corellium was the iOS builder. They may not be related.

@cherrymui cherrymui closed this Dec 9, 2021
@cherrymui cherrymui reopened this Dec 9, 2021
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Dec 9, 2021

Bad click. No intention to close.

@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 9, 2021

Before the "gap" the darwin-arm64-corellium was the iOS builder. They may not be related.

Agreed. (I'm much more concerned about this new one, because I think it's more likely to reflect a source of crashes on real users' hardware.)

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Dec 9, 2021

From the message it seems the bad pointer is 0x14000270000, found at *(0x14000130050+0x8), but later on

*(object+8) = 0x5 <==

It looks like the memory content is changing between the two reads, so there is something concurrently writing to the object. But 5 is also not a valid pointer value. So from the other writer's perspective this field should not be a pointer?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Dec 9, 2021

This may be interesting, or may be completely red-herring:

The object's content looks a lot like some frames' arguments on some of the stacks:

 *(object+0) = 0x140006d4257
 *(object+8) = 0x5 <==
 *(object+16) = 0x16f923847
 *(object+24) = 0x7
 *(object+32) = 0x14000026e20
 *(object+40) = 0x1c
 *(object+48) = 0x1400002a07f
 *(object+56) = 0x10
 *(object+64) = 0x1
 *(object+72) = 0x1

goroutine 262

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4261, 0x3}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 264

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4271, 0x6}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 135

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4151, 0x7}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 266

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4291, 0x7}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

object+16 and later words match exactly with the third and later arguments. object+0 and object+8 are also quite close to the second arguments. However, the object's address, 0x14000130050, doesn't match any of the stacks, and it is on an mSpanInUse span instead of a mSpanManual.

@randall77
Copy link
Contributor

@randall77 randall77 commented Dec 9, 2021

@cherrymui Could it be a layout object (stackArgs?) used by reflect.Call?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Dec 9, 2021

That function is not called by reflect call, so probably not.

@randall77
Copy link
Contributor

@randall77 randall77 commented Dec 9, 2021

Could also be a closure, I guess.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Dec 9, 2021

That call is at https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/load/pkg.go;l=1028
It is in a closure but the closure would capture parent, instead of parent.ImportPath, parent.Dir, parent.Root and other arguments in that order. The order of the fields in the bad object also doesn't match the parent object.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants