testing: fuzzed function can corrupt internal state of fuzzer

[stevenjohnstone]

What version of Go are you using (go version)?

$ go version
go version devel go1.18-d4139083204 Fri Sep 24 07:22:13 2021 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/stevie/.cache/go-build"
GOENV="/home/stevie/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/stevie/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/stevie/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/stevie/sdk/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/stevie/sdk/gotip/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel go1.18-d4139083204 Fri Sep 24 07:22:13 2021 +0000"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/stevie/code/corrupt/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1167654317=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran the following:

package corrupt

import "testing"

func FuzzCorrupt(f *testing.F) {
	f.Fuzz(func(t *testing.T, input []byte) {
		defer func() {
			for i := range input {
				input[i] = 0
			}
		}()

		if len(input) >= 2 && input[0] == 0 && input[1] == 1 {
			t.Fatalf("input %v", input)
		}

	})
}

$ gotip test -fuzz=FuzzCorrupt
warning: starting with empty corpus
fuzz: elapsed: 0s, execs: 0 (0/sec), interesting: 0
fuzz: minimizing 50-byte crash input...
FAIL
fuzz: elapsed: 0s, execs: 745 (16223/sec), interesting: 2
--- FAIL: FuzzCorrupt (0.05s)
        --- FAIL: FuzzCorrupt (0.00s)
            fuzz_test.go:14: input [0 1 0 84 84 84 0 174]
        --- FAIL: FuzzCorrupt (0.00s)
            fuzz_test.go:14: input [0 1 0 84 84 84 0 174]
    
    Crash written to testdata/fuzz/FuzzCorrupt/3be4f06197d4ae9580e884318bf125a616a7cd7eac35346c1e5c943218d8d00f
    To re-run:
    go test corrupt -run=FuzzCorrupt/3be4f06197d4ae9580e884318bf125a616a7cd7eac35346c1e5c943218d8d00f
FAIL
exit status 1
FAIL	corrupt	0.052s
$ cat testdata/fuzz/FuzzCorrupt/3be4f06197d4ae9580e884318bf125a616a7cd7eac35346c1e5c943218d8d00f 
go test fuzz v1
[]byte("\x00\x00\x00\x00\x00\x00\x00\x00")

The "crasher" in the testdata directory does not cause a crash.

Running with minimization disabled also reveals an issue:

$ gotip test -fuzz=FuzzCorrupt -fuzzminimizetime=0
warning: starting with empty corpus
fuzz: elapsed: 0s, execs: 0 (0/sec), interesting: 0
FAIL
FAIL
fuzz: elapsed: 0s, execs: 45 (1354/sec), interesting: 4
--- FAIL: FuzzCorrupt (0.03s)
        --- FAIL: FuzzCorrupt (0.00s)
            fuzz_test.go:14: input [0 1 0 0 0 0 0 0 0 0 0 0 208 230 47 47 47 47 47 47 47 208 208 208 128 0 0]
    
    Crash written to testdata/fuzz/FuzzCorrupt/c0e98f376324bb6d09edf7d1d9d054ca8bd1af158ba4ce46fafd5f96daa76e9f
    To re-run:
    go test corrupt -run=FuzzCorrupt/c0e98f376324bb6d09edf7d1d9d054ca8bd1af158ba4ce46fafd5f96daa76e9f
FAIL
exit status 1
FAIL	corrupt	0.040s
$ gotip test corrupt -run=FuzzCorrupt/c0e98f376324bb6d09edf7d1d9d054ca8bd1af158ba4ce46fafd5f96daa76e9f
ok  	corrupt	0.003s
$ cat testdata/fuzz/FuzzCorrupt/c0e98f376324bb6d09edf7d1d9d054ca8bd1af158ba4ce46fafd5f96daa76e9f 
go test fuzz v1
[]byte("\v\x01\x00\x00\x00G\xea\x10\x00\x01\v\xba\xd0\xe6///////\xd0\xd0Ѐ\x00\xe5")

What did you expect to see?

Crashers stored in testdata should trigger crashes.

What did you see instead?

Corpus entry was not a crasher.

[stevenjohnstone]

Mentioned here #47587 (comment) and https://go-review.googlesource.com/c/go/+/340630/comments/9ad193cf_ca7bf095.

[jayconrod]

cc @golang/fuzzing

[jayconrod]

Allocating a new []byte on every iteration will slow down fuzzing by creating GC pressure. We could keep two slices though and make changes in both: one kept by the fuzzing engine, one passed to the function being fuzzed.

[gopherbot]

Change https://golang.org/cl/353429 mentions this issue: internal/fuzz: prevent the fuzz fn from corrupting data

[katiehockman]

@jayconrod brought up a good point during review of https://golang.org/cl/353429 that makes a case for why we shouldn't be allowing the fuzzed functions to alter the underlying data that is provided by the fuzzing engine. Namely, he brought up that this will be next to impossible when structs are supported in the future. He noted that we would need to allocate a new struct every time the fuzz function is called, which isn't tenable for performance (fuzzing has to be fast to be effective).

One potential option that I brought up is that we support this for []byte only, but not for structs, but special-casing like this makes the code more difficult to maintain and makes things like documentation less straightforward.

This may be something that can be supported in the future after more thought has gone into it, but for now, it sounds like the best course of action is to make it a requirement that the fuzz target is not allowed to alter the underlying data during fuzzing. It's easier to loosen the restriction in future releases then to design ourselves into a corner today.

If folks have good examples of cases where functions like this should be fuzzable, please comment here.