Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: print total number of interesting fuzz inputs, not just the new ones #48669

Closed
mvdan opened this issue Sep 28, 2021 · 3 comments
Closed
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Sep 28, 2021

I ran go test -run=- -fuzz=Foo and within a minute it had reached 600+ interesting inputs; success!

$ go test -run=- -vet=off -fuzz=ParsePrint
fuzz: elapsed: 0s, gathering baseline coverage: 0/2649 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2649/2649 completed, now fuzzing with 16 workers
fuzz: elapsed: 3s, execs: 854257 (284274/sec), interesting: 224
fuzz: elapsed: 6s, execs: 1754833 (292229/sec), interesting: 300
fuzz: elapsed: 9s, execs: 2640098 (293333/sec), interesting: 348
[...]
fuzz: elapsed: 1m51s, execs: 32247392 (290516/sec), interesting: 688
^C

I stopped it via SIGINT, and I was very surprised to see it seemingly start from zero, and much slower:

$ go test -run=- -vet=off -fuzz=ParsePrint
fuzz: elapsed: 0s, gathering baseline coverage: 0/3358 completed
fuzz: elapsed: 0s, gathering baseline coverage: 3358/3358 completed, now fuzzing with 16 workers
fuzz: elapsed: 3s, execs: 940698 (313490/sec), interesting: 1
fuzz: elapsed: 6s, execs: 1938120 (322980/sec), interesting: 1
fuzz: elapsed: 9s, execs: 2900644 (322271/sec), interesting: 2
fuzz: elapsed: 12s, execs: 3853208 (321085/sec), interesting: 2
fuzz: elapsed: 15s, execs: 4818973 (321249/sec), interesting: 4
fuzz: elapsed: 18s, execs: 5777755 (320916/sec), interesting: 5

Note how the first run also caught new interesting inputs much quicker, adding dozens for each log line.

This had me stumped for a while, until I reached into the internal code, and saw that interesting: %d actually stands for the newly discovered intersting inputs, not the grand total. And that interesting inputs are kept in the build cache.

All of that makes complete sense now, but I think the output lines are a bit misleading. I think the following lines would have been far more intuitive (making up the grand total here):

fuzz: elapsed: 3s, execs: 940698 (313490/sec), new interesting: 1 (total: 3321)
fuzz: elapsed: 6s, execs: 1938120 (322980/sec), new interesting: 1 (total: 3321)
fuzz: elapsed: 9s, execs: 2900644 (322271/sec), new interesting: 2 (total: 3322)

cc @katiehockman @jayconrod

@mvdan mvdan added the fuzz Issues related to native fuzzing support label Sep 28, 2021
@jayconrod jayconrod added the NeedsFix The path to resolution is known, but the work has not been done. label Sep 28, 2021
@jayconrod jayconrod added this to the Go1.18 milestone Sep 28, 2021
@ameowlia
Copy link
Contributor

🙋‍♀️ I can do this!

@katiehockman katiehockman added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsFix The path to resolution is known, but the work has not been done. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 29, 2021
@katiehockman
Copy link
Contributor

@ameowlia Sure feel free to send over a PR. Thanks!

@gopherbot
Copy link

Change https://golang.org/cl/353172 mentions this issue: internal/fuzz: print total num of interestings

@golang golang locked and limited conversation to collaborators Oct 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Projects
Status: No status
Development

No branches or pull requests

5 participants