runtime: fatal error: unknown caller pc (libfuzz)

[klauspost]

What version of Go are you using (go version)?

go1.13.3.linux.amd64

Compiled through go-fuzz/libfuzz

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Fuzz test run on https://fuzzit.dev/

Linux, AMD64 is as much as I know

What did you do?

I have 3 crashes crash from "fuzzit.dev" where I am running continuous fuzz testing of my compression packages. Go 1.12.10 was used for 2 builds, Go 1.13.3 for one.

There is no assembly or "unsafe" involved so there shouldn't be any reasonable way for memory corruption. This fuzzing is also strictly running in a single goroutine, so races also seems unlikely.

That said I have no idea about the hardware stability of the servers running the tests.

Also, a lot of new code has just been added here, so there is a chance of something bad, though I don't know how I would be able to trigger this error.

Crash logs: https://gist.github.com/klauspost/d4ec7bd6ecefa1bec56dd8ca4ac8ec39

Go 1.12.10 on top and bottom. Go 1.13.3 in the middle.

It is completely different functions that were pre-empted (flate.(*fastGen).matchlenLong) vs. flate.(*decompressor).Read - completely different code. All crashes were in mgcmark.go:711. Final crash was while executing bytes.(*Buffer).grow.

Crashes have not reproduced locally, so this could be a libfuzz specific problem. Build script is here: https://github.com/klauspost/compress/blob/master/fuzzit.sh#L17 - all crashes have been in the same fuzzer (flate), so it seems something in there is triggering it.

What did you expect to see?

No crash, or more information.

What did you see instead?

Crash.

[klauspost]

This does not use unsafe and the only assembler would be in the stdlib. Imports here: https://godoc.org/github.com/klauspost/compress/flate?imports

"sync" is only used for a sync.Once and there are no goroutines, so no races should happen either.

Fuzz test imports: https://godoc.org/github.com/klauspost/compress-fuzz/flate?imports

This unfortunately happens completely at random.

[aclements]

Just checking if this is related to #35777. What kernel version are you running? Does the application itself receive a lot of signals?

[klauspost]

I don't have access to the servers, so I cannot tell you. It is running on https://fuzzit.dev/ - https://twitter.com/fuzzitdev

[klauspost]

As Keith Randall noted on the original issue #20846

This looks like the stack has been trashed somehow.
Not only the return address for gopark. gopark's arguments also look trashed. The gcBgMarkWorker failure looks similar, hard to tell for sure if its args are trashed as it has only one arg.

Not sure what might cause this. Could be misuse of unsafe, could be runtime bug (use after free of stack memory?).

[prestonvanloon]

Same issue in go1.14, more info about our case here: prysmaticlabs/prysm#5131

[prestonvanloon]

I thought this might be resolved by #37782 in go 1.14.1, but we saw the issue again last night with libfuzzer on go 1.14.1.

[prestonvanloon]

I was able to reproduce locally on kernel 5.3.0-45-generic