cmd/vet: enable fuzz checks in 'tests' analysis pass

[katiehockman]

There are several things that vet can do to support native fuzzing. This issue tracks all of the potential checks that could be added.

Vet should fail if...

• the inputs to any f.Add call don't match the ones in f.Fuzz
• there is any code after f.Fuzz
• the f.Fuzz function is missing a *testing.T as its first parameter
• there is any call to a *testing.F method from within the f.Fuzz function

[timothy-king]

Maybe a dumb question, but can "the f.Fuzz function is missing a *testing.T as its first parameter" be enforced by the typechecker/compiler instead?

[zpavlinovic]

There are several things that vet can do to support native fuzzing. This issue tracks all of the potential checks that could be added.

Vet should fail if...

• the inputs to any f.Add call don't match the ones in f.Fuzz

Could you perhaps elaborate on this one? Do you have a specific pattern in mind?

[katiehockman]

Maybe a dumb question, but can "the f.Fuzz function is missing a *testing.T as its first parameter" be enforced by the typechecker/compiler instead?

@timothy-king Not a dumb question! I don't have an answer to this question though. Possibly? The use of generics might make things easier here as well, but I haven't looked too deeply into this yet.
However, in general, the more issues like this that we can detect at compile-time the better.

Could you perhaps elaborate on this one? Do you have a specific pattern in mind?

@zpavlinovic Yes, I'm thinking of something like this:

func FuzzFoo(f *testing.F) {
  f.Add("some string")
  f.Fuzz(func(*testing.T, int) { })
}

In the case above, f.Add is attempting to add a value to the corpus which is a single string. However, the f.Fuzz function accepts an int as its type. It would be nice if this can fail at compile time, but a vet check would be helpful if that's not feasible. That code should be corrected to something like this:

func FuzzFoo(f *testing.F) {
  f.Add(50)
  f.Fuzz(func(*testing.T, int) { })
}

or

func FuzzFoo(f *testing.F) {
  f.Add("some string")
  f.Fuzz(func(*testing.T, string) { })
}

[guodongli-google]

All these four checks can be covered by a simple vet checker. My main question is whether this checker mets the frequency requirement. After all, running a malformed target will result in panic, so there may be few malformed targets in the tested code.

I am more interested in the bugs that fail "silently", e.g. the expected mutations are not performed. Are there some patterns that are malformed but the related tests won't crash?

[jayconrod]

For anyone looking at this later, golang.org/x/tools/go/analysis/passes/tests is a vet analysis that detects problems with tests, examples, and benchmarks, for example, a missing *T parameter, or an // Output comment in the wrong place.

It's probably a good place to check fuzz tests here, rather than defining a new pass.

[findleyr]

@ansaba is going to look at this. Discussed with @timothy-king and we think it makes sense to start by adding this behavior to gopls alone, then migrate it to vet for 1.19, given that we're in the freeze.

We're not yet sure the best way to stage this; we could use a separate analyzer, or include it in the tests analyzer hidden behind a flag.