Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: support naming seed corpus values provided with (*testing.F).Add #50456

Open
katiehockman opened this issue Jan 5, 2022 · 4 comments

Comments

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jan 5, 2022

Seed corpus entries can be added to a fuzz test by calling (*testing.F).Add. There is no way to name these seed corpus entries, so they default to "seed#{num}". When run with go test -v, it will look something like this:

--- PASS: FuzzFoo (0.00s)
    --- PASS: FuzzFoo/seed#0 (0.00s)
    --- PASS: FuzzFoo/seed#1 (0.00s)
    --- PASS: FuzzFoo/seed#2 (0.00s)

The proposal is to allow a way to name these entries, much like you would with an execution of t.Run. This would help with error messages and debugging. We could do this a few different ways:

  1. Amend f.Add to accept an optional first/last name string param. Since f.Add takes a []interface{} today, this is functionality that could be added later that wouldn't break backwards compatibility.

e.g. existing fuzz tests that look like this:

func FuzzFoo(f *testing.F) {
	f.Add("a3g1f3", 10)
	f.Add("----", 100)
	f.Add("", 0)
	f.Fuzz(func(*testing.T, string, int) {})
}

would change to

func FuzzFoo(f *testing.F) {
	f.Add("valid", "a3g1f3", 10)
	f.Add("invalid", "----", 100)
	f.Add("empty", "", 0)
	f.Fuzz(func(*testing.T, string, int) {})
}
  1. We could add another method on *testing.F. For example:
    (*testing.F).AddNamed(string, []interface{})

Originally proposed by @dnwe

/cc @golang/fuzzing

@thepudds
Copy link

@thepudds thepudds commented Jan 5, 2022

CC @dnwe

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Jan 5, 2022
@mvdan
Copy link
Member

@mvdan mvdan commented Jan 6, 2022

  1. Since f.Add takes a []interface{} today, this is functionality that could be added later that wouldn't break backwards compatibility.

While this will work just fine for the call sites, it would make for a pretty tricky signature: args ...any, but if it begins with an extra string, that's actually the name.

It would be a lot saner if we did name string, args ...any. Is it too late for that? beta1 already shipped, but rc1 hasn't yet, and I don't imagine a lot of people have written native fuzzers yet. We could also leverage go fix to automatically fix fuzz funcs, or supply a gofmt -r expression that people can run.

@mvdan
Copy link
Member

@mvdan mvdan commented Jan 6, 2022

2. (*testing.F).AddNamed(string, []interface{})

For completeness: I realise this is also an option, but leaving Add around as an extra is equally unfortunate, I'd say :) Having just Add(name string, args ...any) feels like the best result overall, especially as it mimics other methods like Run.

@rsc rsc moved this from Incoming to Active in Proposals Jan 12, 2022
@rsc
Copy link
Contributor

@rsc rsc commented Jan 12, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants