-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
testing: minimization takes longer than expected for very large []byte/string #48820
Comments
I figured out what's happening. For the example I provided, the crash only occurs if the []byte is at least 10,000 bytes. The original crash is 34,995 bytes long, which takes a long time to minimize. The first two minimization efforts (cutting the tail and removing every byte) run pretty quickly. However, removing every subset of bytes takes a very long time if the input is long. If I am patient and just let it run for a while, then it will eventually hit that 1 minute minimization timeout and stop gracefully. The part that I found confusing is that it continues to print these "fuzz: elapsed:" messages after minimization has begun, which is a bit jarring and makes it look like the process is hanging. We should probably either 1) stop printing these log messages when we're minimizing or 2) update them to indicate the minimization progress. For (2) that could mean something like "fuzz: elapsed: 12s, minimizing" |
(2) is much better, so the user knows it's not hung. If you were confused, they will be confused. We should also probably do something about the removing subsets of bytes taking forever, but let's handle 'taking longer than user expected' gracefully first. |
Yep agreed. I have a patch ready, and will send that shortly. |
Change https://golang.org/cl/354371 mentions this issue: |
Previously, when fuzzing for a period of time, the command line output would look something like this: fuzz: minimizing 34995-byte crash input... fuzz: elapsed: 3s, execs: 13821 (4604/sec), new interesting: 0 (total: 1) fuzz: elapsed: 6s, execs: 13821 (2303/sec), new interesting: 0 (total: 1) fuzz: elapsed: 9s, execs: 13821 (1535/sec), new interesting: 0 (total: 1) --- FAIL: FuzzFoo (9.05s) This is the same output it has while fuzzing, so if minimization runs for a long time (default allows 1 minute), then it looks like minimization is hanging. It's also confusing that the execs/sec would continually decrease. Now, when minimization is running, the command line output will look something like this: fuzz: minimizing 34995-byte crash input... fuzz: elapsed: 3s, minimizing fuzz: elapsed: 6s, minimizing fuzz: elapsed: 9s, minimizing fuzz: elapsed: 9s, minimizing --- FAIL: FuzzFoo (9.05s) The final "fuzz: elapsed: 6s, minimizing" could be printed twice because we always print one final log to the command line before we exit. Updates #48820 Change-Id: Ie5b9fde48b8d4e36e13a81ae50a6d69bf4d0dbe3 Reviewed-on: https://go-review.googlesource.com/c/go/+/354371 Trust: Katie Hockman <katie@golang.org> Run-TryBot: Katie Hockman <katie@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Moving this to the backlog for now, since at least (2) is done. |
To reproduce, fuzz the following target with
go test -fuzz Fuzz
The following output will occur:
Notice that it found a crash and indicated that it would minimize it, but
then just proceeded with fuzzing anyway(Correction: It didn't keep fuzzing, it just kept updating the log. It looks like the worker hung). If you press Ctrl^C, it exits and writes the crasher:Turning off minimization fixes this (e.g.
go test -fuzz Fuzz -fuzzminimizetime=0x
)This also only seems to happen once the fuzzing engine has already run baseline coverage or found a new interesting value. If the target is changed to panic for every input, this bug won't occur and it'll crash as expected. e.g.
This also shows a hole in our testing. Most of our tests use
-fuzztime
in order to ensure they don't run for too long. However, using-fuzztime
ends fuzzing after a certain number of iterations. That means we aren't testing that the fuzzing process will complete successfully if a crash occurs without help from-fuzztime
./cc @golang/fuzzing
The text was updated successfully, but these errors were encountered: