cmd/go: verify go test flags with -fuzz

[katiehockman]

This is more of a tracking issue, just to have a record of how go test -fuzz behaves when also passed other flags. Let's make sure this is reasonable for the upcoming release for now, and see if we can determine what it would should look like in the long term.

For 1.18, the bare minimum is that it doesn't break anything :)

• -cover, -covermode, -coverpkg

With -run the coverage is printed as zero. We can probably just document that it isn't supported.

godev test -fuzz Fuzz -run Fuzz -cover
...
PASS
coverage: 0.0% of statements
ok      crypto/dsa      3.551s

Without -run the coverage is shown for the other tests that were run. That's good, but I don't want people to get the impression that this coverage includes what was found during fuzzing. Maybe we should print a warning if -cover is provided with the -fuzz flag to indicate that the coverage % doesn't include anything from fuzzing.

godev test crypto/dsa -fuzz FuzzFoo -cover                     
...
PASS
coverage: 88.3% of statements
ok      crypto/dsa      29.526s

• -cpu, -failfast, -short, -shuffle, -timeout, -vet

These simply don't affect fuzzing right now, and maybe never should. We should either document that, add a warning to the command line before fuzzing starts, or both.

• -json, -list, -parallel, -run, -v, -trace benchmarking

Works as intended.

• -mutexprofile

Not 100% sure if this is working as intended, but it at least isn't break anything.

[bcmills]

-short seems like it could affect fuzzing, right? (If a fuzz test calls the testing.Short() function, will it panic?)

It seems like -cpu could affect fuzzing too (by changing runtime.GOMAXPROCS(0)).

-failfast and -vet should already work in combination with fuzzing (should be completely orthogonal, IIUC).

-shuffle should probably shuffle the order of test inputs run from the existing corpus.

[matloob]

Closing this issue since fuzzing has shipped.