testing: document file format for fuzz corpus files

[rogpeppe]

commit 8ac5cbe

The fuzz docs inside the testing package say:

Seed inputs may be registered by calling F.Add or by storing files in the directory testdata/fuzz/ (where is the name of the fuzz target) within the package containing the fuzz target.

but the docs don't say what format those files are expected to be in. I was expecting to be able to just copy a load of corpus data from the original fuzz tool (where the format is just exactly the data for the input), but I see errors of the form:

 "testdata/fuzz/FuzzDecoder/27f4f596adb49c6967f41e8f82d3855f5ae9375e": unmarshal: must include version and at least one value

The testing docs should say what format should be used for these files. Ideally there would be a some kind of migration tool to migrate from the old format to the new.

[mvdan]

I'm also interested in this for a slightly different reason. Imagine I change my fuzzer to add an extra input parameter: presumably all existing fuzz corpus files won't work out of the box anymore, as the number of inputs mismatches.

It would be nice to be able to write a small program (or use a tool) to rewrite the corpus files accordingly, e.g. by setting the new input parameter to its default value.

[mknyszek]

CC @katiehockman

[katiehockman]

You're in luck! @rolandshoemaker already built a tool for this: https://github.com/golang/tools/tree/master/cmd/file2fuzz
We just need to add documentation about the existence of this tool :)

/cc @golang/fuzzing

[rogpeppe]

@katiehockman Thanks! I actually built a little tool myself, but after doing that, I speculated somewhat on what a more general tool might look like. This is what I came up with:

# add an entry to the corpus; the arguments are parsed using the types
# of the function passed f.Fuzz function. If there's no -f argument
# and there's only one Fuzz* function, it'll be implied from the name of
# that. All go-fuzz-corpus commands accept a -f argument.
go-fuzz-corpus -f FuzzFoo add foo bar 1234

# add -file adds all the files as []byte entries. If there
# are any other arguments to the fuzz function, they'll be zero.
# If there's not not exactly one []byte argument to the
# fuzz function, it'll error.
go-fuzz-corpus add -file oldcorpus/*

# migrate migrates old corpus data to new. The argument is a tuple
# expression where "zero" implies the zero value for a type.
# If there aren't enough members of the tuple, additional zero
# members are implied, so an extra argument can be added
# by running migrate with no arguments.
go-fuzz-corpus migrate 'MyStruct{X: old[0], Y: old[1]}, int64(old[2])'

# list lists all items in the corpus, one per line
go-fuzz-corpus list -n

# type prints the type of arguments stored in the corpus.
go-fuzz-corpus type

# check checks that all the entries in the corpus match the expected
# fuzz function. It also warns when there are files that look like corpus
# entries but that have no corresponding Fuzz entry point.
go-fuzz-corpus check

ISTM it might be useful to add something like this to the go command.
I think that it's OK to assume without significant loss of generality that the fuzz types can be inferred directly from the code, which simplifies the design quite a bit. It wouldn't be too hard to add support for an explicit -types flag when people are using reflection or dynamically determined types.

[gopherbot]

Change https://golang.org/cl/372434 mentions this issue: _content/doc/fuzz: document corpus file format