Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: add testing.F.Corpus to help with cross tests conversion #65253

Open
Jorropo opened this issue Jan 24, 2024 · 2 comments
Open
Labels
Milestone

Comments

@Jorropo
Copy link
Member

Jorropo commented Jan 24, 2024

Proposal Details

I have two fuzz tests which test the same code, however one of them accepts binary data and the second one takes strings (my code has an ASCII and binary representation).

I was able to find new issues by seeding the corpus from the other test by converting the representation.

So I propose this to be added:

// Corpus let you fetch the corpus from an other test, callback will be called with all the entry in the other test corpus.
// source must be a function pointer to the actual other test.
// callback must be a function pointer which has the same signature as the other test [F.Add] argument.
// Corpus must not be called once [F.Fuzz] has been called.
func (*F) Corpus(source, callback any)

Expected usage:

func FuzzString(f *testing.F) {
 for _, v := range testcase {
  _, err := decodeString(v)
  if err != nil { f.Fatal(err) }
  f.Add(v)
 }
 f.Corpus(FuzzBinary, func(_ *testing.T, b []byte) {
  v, err := decodeBinary(b)
  if err != nil { return }
  str, err := v.String()
  if err != nil { return }
  f.Add(str)
 })
 f.Fuzz(func(_ *testing.T, s string) {
  v, err := decodeString(s)
  if err != nil { return }
  testStuff(v)
 })
}

func FuzzBinary(f *testing.F) {
 for _, v := range testcase {
  b, err := decodeString(v)
  if err != nil { f.Fatal(err) }
  f.Add(b)
 }
 f.Corpus(FuzzString, func(_ *testing.T, s string) {
  v, err := decodeString(s)
  if err != nil { return }
  f.Add(v)
 })
 f.Fuzz(func(_ *testing.T, b []byte) {
  v, err := decodeBinary(b)
  if err != nil { return }
  testStuff(v)
 })
}

It's unclear to me if callback should be retained, if it is retained we could run multiple fuzz tests in parallel and convert entries from one into one from the other on the fly (it would require some tricky-ish synchronisation and validation internally).
I think for now this is not needed but we can add a comment hinting this might be a thing in the future. I don't think this would break anyone since you can't currently run more than one fuzz test together.

@gopherbot gopherbot added this to the Proposal milestone Jan 24, 2024
@Jorropo
Copy link
Member Author

Jorropo commented Jan 24, 2024

I think it should also be legal to fetch your own corpus (altho a bit pointless).
This is so if we have 10 interconnected fuzz test we don't need 10 snowflake fetch conversion.
I could have a single corpus(*testing.F) function which fetch and add the corpus from all the 10 tests.

@ianlancetaylor ianlancetaylor moved this to Incoming in Proposals Feb 7, 2024
@ianlancetaylor
Copy link
Member

CC @golang/fuzzing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

3 participants