testing: sub fuzz targets

[srikrsna]

It would be nice to be able to define sub fuzz targets like subtests and sub-benchmarks. Example:

for _, fc := range Cases {
    f.Run(fc.Name, func(f *testing.F) {
        f.Add(...)
        f.Fuzz(func(t *testing.T, data []byte) {
            t.Parallel()
            fz := fuzz.NewFromGoFuzz(data).Funcs(exfuzz.FuzzFuncs()...).Funcs(vanguardfz.FuzzFuncs()...)

            *r := fc.r
            fz.Fuzz(&r)

            u := fc.Permissions
            res, det, err := assert[tc.Method].Eval(map[string]interface{}{
                "r": r,
                "u": u,
            })
            // Assertions
        })
    })
}

This would be very much useful to fuzz functions with generic but defined data structures.

[cagedmantis]

/cc @golang/fuzzing

[jayconrod]

It's not clear to me exactly what this would do or how it would be used. Could you elaborate?

This would be very much useful to fuzz functions with generic but defined data structures.

This is something we'd like to support eventually in any case. You wouldn't need a sub-target for it.

type Foo struct { ... }
func FuzzFoo(f *testing.F) {
  f.Fuzz(func(t *testing.T, foo Foo) {
    // check something with the Foo struct, without having to deal with []byte
  })
}

[rolandshoemaker]

It's not clear to me exactly what this would do or how it would be used.

Agreed, this seems like it'd also only work when -fuzztime is specified, otherwise what is the behavior when the fuzzer is running indefinitely?

This would be very much useful to fuzz functions with generic but defined data structures.

When fuzzing a function that takes an interface{} (i.e. a JSON parser or something), a common approach is to test all data structures in the same fuzz target, rather than using a single target per structure, since they're likely to share significant logic.

func FuzzJSON(f *testing.F) {
	f.Fuzz(func(t *testing.T, b []byte) {
		for _, typ := range []interface{}{
			map[interface{}]interface{}{},
			[]interface{}{},
		} {
			json.Unmarshal(b, typ)
		}
	})
}

[srikrsna]

The use case is similar to table driven tests. As an example, I want to Fuzz test my input validation logic in my gRPC service. In the current approach I will have to write a top level Fuzz target for each of the request methods.

In case of sub-targets, I would write it something like this,

func FuzzValidate(f *testing.F) {
   ff := []struct{
       Name string
       Seed proto.Message
       Request proto.Message
   }{
      {...},
      {...},
       ...
   }

   for _, fc := range ff {
    f.Run(fc.Name, func(f *testing.F) {
        // f.Parallel() to solve the fuzztime requirement
        f.Fuzz(func(t *testing.T, data []byte) {
            t.Parallel()
            fz := fuzz.NewFromGoFuzz(data)
            *r := fc.r
            fz.Fuzz(&r)

            r.Validate()            
        })
    })    
   }   
}

If we were to shift the loop inside, I wouldn't know which validation logic is failing.

[jayconrod]

@srikrsna Thanks, I think that makes sense.

We'd need to nail down the exact semantics for this.

• It should not be allowed to call F.Run and F.Fuzz on the same F, just as it's not allowed to call F.Fuzz more than once.
• We'd probably want functions that call F.Run to finish before starting any function provided to F.Fuzz (which already runs in a goroutine). That way, we can ensure -test.fuzz matches exactly one target.

[srikrsna]

My assumption is that the semantics would be similar to benchmarks, won't they?

[rolandshoemaker]

If we were to shift the loop inside, I wouldn't know which validation logic is failing.

It seems like manual testing of the triggering input would give you this information, which is generally doing to be required anyway when a new crasher is found (additionally the trace should point you to where the problem is occurring).

[srikrsna]

additionally the trace should point you to where the problem is occurring

The same applies for sub tests and sub benchmarks. Trace would useful to find out the exact line where the problem would be and how it can be fixed. Naming the targets will give an overview of which cases are failing.

[AlekSi]

f.Add would greatly help in understanding the log of the failed test:

func FuzzFoo(f *testing.F) {
	for _, tc := range testcases {
		tc := tc
		f.Run(tc.name, func(t *testing.T) { // note
			t.Parallel()

			// Test with tc.input.
			// If test failes, tc.name will be logged by f.Run.

			f.Add(tc.input)
		})
	}

	f.Fuzz(func(t *testing.T, input InputType) {
		// fuzz
	})
}

Note that f.Run could accept function accepting testing.T instead of testing.F. That way, there is only one testing.F, and less ways to accidentally call f.Fuzz.

[AlekSi]

@gopherbot fuzz