Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: unexpected fault address 0 #67255

Closed
karalabe opened this issue May 8, 2024 · 13 comments
Closed

runtime: unexpected fault address 0 #67255

karalabe opened this issue May 8, 2024 · 13 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FixPending Issues that have a fix which has not yet been reviewed or submitted. NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@karalabe
Copy link
Contributor

karalabe commented May 8, 2024

Go version

go1.22.3 linux-amd64

Output of go env in your module/workspace:

Hard to say exactly, I use `golang:1.22-alpine` (latest) docker container to build a Go app and add it to an `alpine:latest` docker container to run.

What did you do?

I can reproduce this consistently (i.e. happens 100% of the time), but it is extremely deep in a live system, so I have no idea how to make a proper repro out of it.

I have the following Go code crashing:

		values := someFunc(...)

		values = append(make([][]byte, first), values...)
		values = append(values, make([][]byte, len(keys)-last)...)

The crash is in the runtime memory management. With Go 1.22.2 I got the following dump:

	unexpected fault address 0x0
	fatal error: fault
	[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x41b0f7]
	goroutine 226018 gp=0xc000e1f500 m=32 mp=0xc0079b1008 [running]:
	runtime.throw({0x1a826de?, 0xc047f3fcb8?})
		runtime/panic.go:1023 +0x5c fp=0xc047f3fc58 sp=0xc047f3fc28 pc=0x43fc3c
	runtime.sigpanic()
	    runtime/signal_unix.go:895 +0x285 fp=0xc047f3fcb8 sp=0xc047f3fc58 pc=0x4588e5
	runtime.(*mspan).typePointersOfUnchecked(0x0?, 0x0?)
	    runtime/mbitmap_allocheaders.go:202 +0x37 fp=0xc047f3fcd8 sp=0xc047f3fcb8 pc=0x41b0f7
	runtime.(*mspan).typePointersOf(0x7f20aca3af30, 0xc047ceca00, 0x0)
	    runtime/mbitmap_allocheaders.go:162 +0x45 fp=0xc047f3fd20 sp=0xc047f3fcd8 pc=0x41b065
	runtime.bulkBarrierPreWrite(0xc047ceca00, 0x0, 0x0, 0x35?)
	    runtime/mbitmap_allocheaders.go:465 +0x12d fp=0xc047f3fd90 sp=0xc047f3fd20 pc=0x41b52d
	runtime.memclrHasPointers(0xc047ceca00, 0x0)
	    runtime/mbarrier.go:371 +0x25 fp=0xc047f3fdc0 sp=0xc047f3fd90 pc=0x41a1e5
        [...]
        at this point the line is the last line from my snippet above

With Go 1.22.3 I still see the exact crash but it didn't spit out this much crash dump, only a shorter one (which 1.22.2 also spits out sometimes instead of the long one)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x41b0f7]
goroutine 115123 [running]:
github.com/ethereum/go-ethereum/trie.(*Trie).getBatch(0xc03ad422d0, {0x2009770?, 0xc0489ab450}, {0xc0488daa88, 0x25, 0x25}, 0x7)
 	github.com/ethereum/go-ethereum/trie/trie.go:286 +0xf65

The line referenced is the same exact last line the previous crash references; a simple slice append.

What did you see happen?

Boom

What did you expect to see?

No boom

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label May 8, 2024
@karalabe
Copy link
Contributor Author

karalabe commented May 8, 2024

For what it's worth, the boom seems to happen with first == 0; len(keys) == len(values) == N and last == n, so something akin to:

	values := f(keys)

	values = append(make([][]byte, 0), values...)
	values = append(values, make([][]byte, 0)...)

Obviously I can't make Go crash on such a simple repro :)) Something needs to bork within the garbage collector first I assume.

@karalabe
Copy link
Contributor Author

karalabe commented May 8, 2024

Managed to blow it up verbose with Go 1.22.3 too:

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x41b0fb]
goroutine 876768 gp=0xc0009e8c40 m=23 mp=0xc000981808 [running]:
runtime.throw({0x1a826de?, 0xc06e640148?})
	runtime/panic.go:1023 +0x5c fp=0xc06e6400e8 sp=0xc06e6400b8 pc=0x43fc3c
runtime.sigpanic()
	runtime/signal_unix.go:895 +0x285 fp=0xc06e640148 sp=0xc06e6400e8 pc=0x4588e5
runtime.readUintptr(...)
	runtime/mbitmap.go:466
runtime.(*mspan).typePointersOfUnchecked(0x0?, 0x0?)
	runtime/mbitmap_allocheaders.go:203 +0x3b fp=0xc06e640168 sp=0xc06e640148 pc=0x41b0fb
runtime.(*mspan).typePointersOf(0x7f0d81965270, 0xc07403a2c0, 0x0)
	runtime/mbitmap_allocheaders.go:162 +0x45 fp=0xc06e6401b0 sp=0xc06e640168 pc=0x41b065
runtime.bulkBarrierPreWrite(0xc07403a2c0, 0x0, 0x0, 0x1d?)
	runtime/mbitmap_allocheaders.go:465 +0x12d fp=0xc06e640220 sp=0xc06e6401b0 pc=0x41b52d
runtime.memclrHasPointers(0xc07403a2c0, 0x0)
	runtime/mbarrier.go:371 +0x25 fp=0xc06e640250 sp=0xc06e640220 pc=0x41a1e5
github.com/ethereum/go-ethereum/trie.(*Trie).getBatch(0xc06fb279e0, {0x2009730?, 0xc073ba11d0}, {0xc07402e588, 0x1d, 0x1d}, 0x7)
	github.com/ethereum/go-ethereum/trie/trie.go:285 +0xe93 fp=0xc06e640410 sp=0xc06e640250 pc=0xd0c693

Mostly the same crash, just one line difference in mbitmap_allocheaders

@mauri870
Copy link
Member

mauri870 commented May 8, 2024

Without a repro these issues are hard to investigate. Are you sure that running with -race does not point any issues? Thanks.

@mauri870 mauri870 added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 8, 2024
@randall77
Copy link
Contributor

I think I see the bug here. Simplified:

func g(x []*byte, z int) []*byte {
	return append(x, make([]*byte, z)...)
}

When we call this with z==0 and x is a slice that points to an object that completely fills its size class, then we call memclrHasPointers with x.ptr + 8*x.len, 0, which may point to the next object in the span. That is a no-no, as the next object may be junk.

We need some sort of defense against past-the-end pointers in this generated code.

Possibly introduced in CL 418554

@karalabe
Copy link
Contributor Author

karalabe commented May 8, 2024

Ha, reproed it.

https://go.dev/play/p/zfi-jTPIIp_K

Screenshot 2024-05-08 at 17 46 16

@mauri870 mauri870 added NeedsFix The path to resolution is known, but the work has not been done. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 8, 2024
@karalabe
Copy link
Contributor Author

karalabe commented May 8, 2024

Further simplified: https://go.dev/play/p/pLh04Z_lzSd

Screenshot 2024-05-08 at 17 49 47

@karalabe
Copy link
Contributor Author

karalabe commented May 8, 2024

Interestingly, with 100 items in the slice, it doesn't crash, with 53, it does. I'm assuming the former expands the array "nicely" whereas the latter hits some cornercase.

@karalabe
Copy link
Contributor Author

karalabe commented May 8, 2024

PS: The reason I pasted the images and not just stack traces is because I'm unsure how consistent the crashes are, so figured I'd better make some "proof".

@karalabe
Copy link
Contributor Author

karalabe commented May 8, 2024

Got a scarier one now: https://go.dev/play/p/AbxNigAFhCk

Screenshot 2024-05-08 at 18 03 35

@randall77
Copy link
Contributor

@gopherbot please open a backport issue for 1.22.

@gopherbot
Copy link
Contributor

Backport issue(s) opened: #67258 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@randall77
Copy link
Contributor

Seems to be ok in 1.21 and earlier.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/584116 mentions this issue: cmd/compile: avoid past-the-end pointer when zeroing

@dmitshur dmitshur added this to the Go1.23 milestone May 8, 2024
@dmitshur dmitshur added the FixPending Issues that have a fix which has not yet been reviewed or submitted. label May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FixPending Issues that have a fix which has not yet been reviewed or submitted. NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

5 participants