proposal: testing: support naming seed corpus values provided with (*testing.F).Add

[katiehockman]

Seed corpus entries can be added to a fuzz test by calling (*testing.F).Add. There is no way to name these seed corpus entries, so they default to "seed#{num}". When run with go test -v, it will look something like this:

--- PASS: FuzzFoo (0.00s)
    --- PASS: FuzzFoo/seed#0 (0.00s)
    --- PASS: FuzzFoo/seed#1 (0.00s)
    --- PASS: FuzzFoo/seed#2 (0.00s)

The proposal is to allow a way to name these entries, much like you would with an execution of t.Run. This would help with error messages and debugging. We could do this a few different ways:

1. Amend f.Add to accept an optional first/last name string param. Since f.Add takes a []interface{} today, this is functionality that could be added later that wouldn't break backwards compatibility.

e.g. existing fuzz tests that look like this:

func FuzzFoo(f *testing.F) {
	f.Add("a3g1f3", 10)
	f.Add("----", 100)
	f.Add("", 0)
	f.Fuzz(func(*testing.T, string, int) {})
}

would change to

func FuzzFoo(f *testing.F) {
	f.Add("valid", "a3g1f3", 10)
	f.Add("invalid", "----", 100)
	f.Add("empty", "", 0)
	f.Fuzz(func(*testing.T, string, int) {})
}

2. We could add another method on *testing.F. For example:
   (*testing.F).AddNamed(string, []interface{})

Originally proposed by @dnwe

/cc @golang/fuzzing

[thepudds]

CC @dnwe

[mvdan]

1. Since f.Add takes a []interface{} today, this is functionality that could be added later that wouldn't break backwards compatibility.

While this will work just fine for the call sites, it would make for a pretty tricky signature: args ...any, but if it begins with an extra string, that's actually the name.

It would be a lot saner if we did name string, args ...any. Is it too late for that? beta1 already shipped, but rc1 hasn't yet, and I don't imagine a lot of people have written native fuzzers yet. We could also leverage go fix to automatically fix fuzz funcs, or supply a gofmt -r expression that people can run.

[mvdan]

2. (*testing.F).AddNamed(string, []interface{})

For completeness: I realise this is also an option, but leaving Add around as an extra is equally unfortunate, I'd say :) Having just Add(name string, args ...any) feels like the best result overall, especially as it mimics other methods like Run.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

It's getting very late to change the API, so it's looking like AddNamed is the right approach?
Or maybe we don't need this feature at all?

Another possibility is to define a new string type like testing.FuzzName and then you could

f.Add(testing.FuzzName("myname"), "data")

but that's a bit verbose.

[mvdan]

I reckon we're not late to change the API; this user report was given three weeks ago, a couple of weeks after the beta came out. That's very much within the timeframe that you would expect users to give the beta an honest try and report issues. At the time of writing there are 38 open issues with the release-blocker label, so I also reckon the RC is at least a couple of weeks away, and shouldn't be further delayed by a minor API change.

I do think we need this feature; if we say "one shouldn't need to name seed corpus entries", presumably the same could have applied to sub-test and sub-benchmark names :) And I still believe that altering Add while we still can is the best and most consistent approach long-term.

[mvdan]

Also, if it comes between AddNamed and FuzzName, my vote would go for FuzzName; that way we avoid two "add" entrypoints, and it's clear which argument is the name versus the variable number of fuzz values.

Finally, just a thought: I seem to recall that the proposal review committee only meets once a week, and the RC isn't very far, so we should probably expedite a resolution here before more weeks go by and we really are too late.

[mvdan]

I also reckon the RC is at least a couple of weeks away

It seems like we'll get beta2 next week, which seems to support my reasoning above - the RC is probably four weeks away or so. And, if we can manage to get this small change into master early next week, users can also test it as part of beta2 :)

[rsc]

I am not sure about the need to name specific seed values. This reminds me of the pets vs cattle discussion for managing computers.

Every test function we write is typically an important, unique, carefully tended thing (like a pet).
I am not convinced fuzzing seed corpus values are like that at all: do we really want the overhead of having to name them?
Seeds are usually small: why not just give the subtest for it a sequence number and then print the actual value when it fails?

We are talking about adding complexity, and I am not convinced we have established that the complexity is needed.
And it is very late in the cycle to add complexity that we're not sure we need.
(Package constraints just missed the cut! Is this really more important than that? And are we so sure?)

[mvdan]

Perhaps you're right that not every seed corpus entry will need a name, but I think that already applies to sub-tests and sub-benchmarks - one can leave the name empty and they get a number instead. That seems like the best of both worlds to me.

Seeds are usually small: why not just give the subtest for it a sequence number and then print the actual value when it fails?

That assumption worries me a little; most of the fuzzers I've written take bytes or strings as input, and some of those inputs do get reasonably long fairly quickly when I want the seed corpus to also cover some particularly interesting and complex edge cases.

That said, if we think that unnamed seed corpus entries will be more common, then perhaps the FuzzName solution is a good middle ground. I can't say I have enough data to point one way or another.

I'm not sure if the comparison with package constraints applies; we can always add that package in 1.19, whereas we can't change the signature of testing.F.Add once it gets released :)

[katiehockman]

whereas we can't change the signature of testing.F.Add once it gets released :)

I don't think that adding the testing.FuzzName type would be a non-backwards compatible change to the signature of testing.F.Add if we were to add this in Go 1.19. Either way, the fuzz name should be optional. Given that, we could make both of the following testing.F.Add calls valid, whereas in Go 1.18 only the first would be:

f.Add("data")
f.Add(testing.FuzzName("myname"), "data")

testing.F.Add takes args ...any today, so we can certainly add an optional additional value later that will be processed as the name, once we've had more time to collect evidence, and when we don't have to rush.