We should support corpus minimization, which will mean the ability to remove items from the on-disk corpus which either 1) don't have the types that are supported by the fuzz test (e.g. they are left over from a previous version of the test which took different params), or 2) don't expand any new coverage that isn't already provided by other entries in the corpus.
(1) should be pretty straightforward. We can just unmarshal the contents of the file, and see if it matches. If it doesn't, delete it.
(2) will be a bit more involved, but not necessarily that complicated. We should take a look at how libFuzzer implements this. One potential solution would be to maintain a coverage map, and run each corpus item against the fuzz test in turn, updating the map is it runs. If any of them don't expand coverage, delete it. A potential pitfall of this: if we have 20 corpus entries that each expand 1 line, and 1 corpus entry that covers all 20 of those lines at once, which do we choose?
At least (2) will be needed for OSS-Fuzz integration, if they end up supporting native support.
The text was updated successfully, but these errors were encountered:
A thought regarding (2): if we have an effective minimization algorithm, is it actually necessary to prune the corpus on disk? It probably is worthwhile to prune the inputs stored in the Go build cache, but arguably the former-crashers in the testdata/fuzz directory should be retained as regression tests.
Perhaps we could instead start each fuzzing run by spending some fraction of the time budget re-minimizing the corpus. That would also avoid losing coverage if (for example) the code is modified in a way that significantly changes coverage before refactoring back to something closer to a previous approach (for which the pruned-out inputs might actually become relevant again).
It probably is worthwhile to prune the inputs stored in the Go build cache, but arguably the former-crashers in the testdata/fuzz directory should be retained as regression tests.
Yes agreed. When I talk about "corpus minimization" here, I'm only referring to the corpus in the build cache. We shouldn't touch the seed corpus.
Perhaps we could instead start each fuzzing run by spending some fraction of the time budget re-minimizing the corpus.
We could do this, but I would argue it should be opt in. e.g. go test -fuzz=Fuzz -fuzzminimizecorpus
Otherwise, I'm imagining a scenario where someone has a fuzz target that accepts byte. In playing around with their test, they change the target to accept byte, int. If we prune the corpus by default, then when they run go test -fuzz Fuzz, all of the cache corpus that only accepts byte is going to be deleted before the user realizes what happened.