Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: found bad pointer in Go heap (1.19.1) #56552

Open
mobiusmalware opened this issue Nov 3, 2022 · 5 comments
Open

runtime: fatal error: found bad pointer in Go heap (1.19.1) #56552

mobiusmalware opened this issue Nov 3, 2022 · 5 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@mobiusmalware
Copy link

mobiusmalware commented Nov 3, 2022

I have a parsing program that's been in development for the past few months to pull a variety of information from a large sample set. The current issue is that, at random, I will get a fatal error that will kill the program when running against any given directory:

runtime: pointer 0xc000d8b750 to unused region of span span.base()=0xc000b12000 span.limit=0xc000b13fe0 span.state=1
runtime: found in object at *(0xc00112a040+0x0)
object=0xc00112a040 s.base()=0xc00112a000 s.limit=0xc00112c000 s.spanclass=8 s.elemsize=32 s.state=mSpanInUse
 *(object+0) = 0xc000d8b750 <==
 *(object+8) = 0xc000f52000
 *(object+16) = 0x1000
 *(object+24) = 0x1000
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x76b22e?, 0x6?})
	/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f882d7f9d00 sp=0x7f882d7f9cd0 pc=0x43871d
runtime.badPointer(0x7f8803efa2a8, 0x0?, 0xc00112a040, 0xd590?)
	/usr/local/go/src/runtime/mbitmap.go:373 +0x150 fp=0x7f882d7f9d50 sp=0x7f882d7f9d00 pc=0x415770
runtime.findObject(0xc00112a040?, 0x9d86e0?, 0x40d049?)
	/usr/local/go/src/runtime/mbitmap.go:416 +0xa6 fp=0x7f882d7f9d88 sp=0x7f882d7f9d50 pc=0x415906
runtime.scanobject(0xc00002f740?, 0xc00002f740)
	/usr/local/go/src/runtime/mgcmark.go:1338 +0x190 fp=0x7f882d7f9e20 sp=0x7f882d7f9d88 pc=0x421490
runtime.gcDrain(0xc00002f740, 0x2)
	/usr/local/go/src/runtime/mgcmark.go:1103 +0x1d4 fp=0x7f882d7f9e80 sp=0x7f882d7f9e20 pc=0x420cd4
runtime.gcBgMarkWorker.func2()
	/usr/local/go/src/runtime/mgc.go:1323 +0x154 fp=0x7f882d7f9ed0 sp=0x7f882d7f9e80 pc=0x41d474
runtime.systemstack()
	/usr/local/go/src/runtime/asm_amd64.s:492 +0x49 fp=0x7f882d7f9ed8 sp=0x7f882d7f9ed0 pc=0x4695a9

The full error can be seen here: https://go.dev/play/p/5m48pLjUsEK

Additionally, this is not intentionally reproducible. Running my program against the same directory again will not yield the error, and it may go through several thousand or several hundred thousand files before it occurs again. I have multiple logs of this activity.

Some things to note:

  • The executable is built with CGO_ENABLED=0 due to compatibility issues with the box it's being run on
  • All drives being scanned are mounted via mount -o ro /remote/drive/path /local/drive/path
  • There is no use of unsafe or cgo in my code; however, there is a dependency for the parser that has a single use. This is located in https://github.com/edsrzf/mmap-go/blob/main/mmap.go (lines 80-82):
func (m *MMap) header() *reflect.SliceHeader {
	return (*reflect.SliceHeader)(unsafe.Pointer(m))
}
  • This has not been an issue previous to the last week of testing, but I have not made any changes to the dependency or parser code in that time.

What could be causing this to occur, and is there a way to correct for it?

go version: 1.19.1 linux/amd64
OS: Ubuntu 22.04.1 LTS
Actions: Ran parsing program against mounted drives (which are also running Linux)
Expected: No pointer failures
Observed: Random pointer failures

@mdempsky
Copy link
Member

mdempsky commented Nov 3, 2022

/cc @golang/runtime

FWIW, I spot checked the mentioned use of unsafe.Pointer in mmap-go/blob/main/mmap.go and it seems okay. As far as I can tell, it's only used on Windows anyway.

@mdempsky mdempsky changed the title Random Pointer Failure Mystery runtime: fatal error: found bad pointer in Go heap (1.19.1) Nov 3, 2022
@mdempsky mdempsky added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 3, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 3, 2022
@randall77
Copy link
Contributor

randall77 commented Nov 3, 2022

Have you tried running it under the race detector?

@mknyszek mknyszek added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 9, 2022
@mknyszek mknyszek added this to the Backlog milestone Nov 9, 2022
@mobiusmalware
Copy link
Author

mobiusmalware commented Nov 14, 2022

@randall77 I've tried with the race detector and had no failures, even against files that were parsed immediately before the pointer failure. That also required me to rebuild the executable with CGO enabled, which changed things.

@joedian joedian removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 15, 2022
@mknyszek
Copy link
Contributor

mknyszek commented Nov 16, 2022

Awesome, thanks for confirming. Without a way to reproduce this it's hard to make progress on our end. It could be a real bug in the GC based on the information you shared. Do you have any additional information, or can you point us to some code we could run that at least might reproduce the issue?

@mknyszek mknyszek added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 16, 2022
@mobiusmalware
Copy link
Author

mobiusmalware commented Nov 16, 2022

@mknyszek I'd love to be able to point you to some code but I need permission to upload, so I'll have to wait for that. What I do have is a saved callstack from this error that I managed to capture via GoLand while remotely debugging this morning. However, running the program against the files to replicate...failed. It's still humming along, hours later, without issue. Which is the major problem in that I can't just reproduce the error at will.
goroutines.txt
Goroutine 37 is where the pointer issue occurs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Status: No status
Development

No branches or pull requests

6 participants