Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: sub fuzz targets #46780

Open
srikrsna opened this issue Jun 16, 2021 · 12 comments
Open

testing: sub fuzz targets #46780

srikrsna opened this issue Jun 16, 2021 · 12 comments

Comments

@srikrsna
Copy link

@srikrsna srikrsna commented Jun 16, 2021

It would be nice to be able to define sub fuzz targets like subtests and sub-benchmarks. Example:

for _, fc := range Cases {
    f.Run(fc.Name, func(f *testing.F) {
        f.Add(...)
        f.Fuzz(func(t *testing.T, data []byte) {
            t.Parallel()
            fz := fuzz.NewFromGoFuzz(data).Funcs(exfuzz.FuzzFuncs()...).Funcs(vanguardfz.FuzzFuncs()...)

            *r := fc.r
            fz.Fuzz(&r)

            u := fc.Permissions
            res, det, err := assert[tc.Method].Eval(map[string]interface{}{
                "r": r,
                "u": u,
            })
            // Assertions
        })
    })
}

This would be very much useful to fuzz functions with generic but defined data structures.

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jun 16, 2021

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jun 21, 2021

It's not clear to me exactly what this would do or how it would be used. Could you elaborate?

This would be very much useful to fuzz functions with generic but defined data structures.

This is something we'd like to support eventually in any case. You wouldn't need a sub-target for it.

type Foo struct { ... }
func FuzzFoo(f *testing.F) {
  f.Fuzz(func(t *testing.T, foo Foo) {
    // check something with the Foo struct, without having to deal with []byte
  })
}

@rolandshoemaker
Copy link
Member

@rolandshoemaker rolandshoemaker commented Jun 21, 2021

It's not clear to me exactly what this would do or how it would be used.

Agreed, this seems like it'd also only work when -fuzztime is specified, otherwise what is the behavior when the fuzzer is running indefinitely?

This would be very much useful to fuzz functions with generic but defined data structures.

When fuzzing a function that takes an interface{} (i.e. a JSON parser or something), a common approach is to test all data structures in the same fuzz target, rather than using a single target per structure, since they're likely to share significant logic.

func FuzzJSON(f *testing.F) {
	f.Fuzz(func(t *testing.T, b []byte) {
		for _, typ := range []interface{}{
			map[interface{}]interface{}{},
			[]interface{}{},
		} {
			json.Unmarshal(b, typ)
		}
	})
}

@srikrsna
Copy link
Author

@srikrsna srikrsna commented Jun 21, 2021

The use case is similar to table driven tests. As an example, I want to Fuzz test my input validation logic in my gRPC service. In the current approach I will have to write a top level Fuzz target for each of the request methods.

In case of sub-targets, I would write it something like this,

func FuzzValidate(f *testing.F) {
   ff := []struct{
       Name string
       Seed proto.Message
       Request proto.Message
   }{
      {...},
      {...},
       ...
   }

   for _, fc := range ff {
    f.Run(fc.Name, func(f *testing.F) {
        // f.Parallel() to solve the fuzztime requirement
        f.Fuzz(func(t *testing.T, data []byte) {
            t.Parallel()
            fz := fuzz.NewFromGoFuzz(data)
            *r := fc.r
            fz.Fuzz(&r)

            r.Validate()            
        })
    })    
   }   
}

If we were to shift the loop inside, I wouldn't know which validation logic is failing.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jun 21, 2021

@srikrsna Thanks, I think that makes sense.

We'd need to nail down the exact semantics for this.

  • It should not be allowed to call F.Run and F.Fuzz on the same F, just as it's not allowed to call F.Fuzz more than once.
  • We'd probably want functions that call F.Run to finish before starting any function provided to F.Fuzz (which already runs in a goroutine). That way, we can ensure -test.fuzz matches exactly one target.

@srikrsna
Copy link
Author

@srikrsna srikrsna commented Jun 21, 2021

My assumption is that the semantics would be similar to benchmarks, won't they?

@rolandshoemaker
Copy link
Member

@rolandshoemaker rolandshoemaker commented Jun 21, 2021

If we were to shift the loop inside, I wouldn't know which validation logic is failing.

It seems like manual testing of the triggering input would give you this information, which is generally doing to be required anyway when a new crasher is found (additionally the trace should point you to where the problem is occurring).

@srikrsna
Copy link
Author

@srikrsna srikrsna commented Jun 28, 2021

additionally the trace should point you to where the problem is occurring

The same applies for sub tests and sub benchmarks. Trace would useful to find out the exact line where the problem would be and how it can be fixed. Naming the targets will give an overview of which cases are failing.

@AlekSi

This comment was marked as off-topic.

@AlekSi

This comment was marked as off-topic.

@gopherbot gopherbot added the fuzz label Jul 26, 2021
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jul 26, 2021

@AlekSi If the goal is to associate a name with each seed value for output, maybe open a new issue? As I understand it, this one is about having separate fuzz targets with their own fuzz separate functions, but perhaps with shared setup code.

@AlekSi
Copy link
Contributor

@AlekSi AlekSi commented Jul 27, 2021

Created #47413 for adding (*F) Run(string, func(t *T)) instead of (*F) Run(string, func(f *F)).

@rsc rsc changed the title [dev.fuzz] testing: sub fuzz targets testing: sub fuzz targets Sep 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants