testing: minimization takes longer than expected for very large byte/string #48820
To reproduce, fuzz the following target with
The following output will occur:
Notice that it found a crash and indicated that it would minimize it, but
Turning off minimization fixes this (e.g.
This also only seems to happen once the fuzzing engine has already run baseline coverage or found a new interesting value. If the target is changed to panic for every input, this bug won't occur and it'll crash as expected. e.g.
This also shows a hole in our testing. Most of our tests use
The text was updated successfully, but these errors were encountered:
I figured out what's happening. For the example I provided, the crash only occurs if the byte is at least 10,000 bytes. The original crash is 34,995 bytes long, which takes a long time to minimize. The first two minimization efforts (cutting the tail and removing every byte) run pretty quickly. However, removing every subset of bytes takes a very long time if the input is long.
If I am patient and just let it run for a while, then it will eventually hit that 1 minute minimization timeout and stop gracefully. The part that I found confusing is that it continues to print these "fuzz: elapsed:" messages after minimization has begun, which is a bit jarring and makes it look like the process is hanging. We should probably either 1) stop printing these log messages when we're minimizing or 2) update them to indicate the minimization progress. For (2) that could mean something like "fuzz: elapsed: 12s, minimizing"
Previously, when fuzzing for a period of time, the command line output would look something like this: fuzz: minimizing 34995-byte crash input... fuzz: elapsed: 3s, execs: 13821 (4604/sec), new interesting: 0 (total: 1) fuzz: elapsed: 6s, execs: 13821 (2303/sec), new interesting: 0 (total: 1) fuzz: elapsed: 9s, execs: 13821 (1535/sec), new interesting: 0 (total: 1) --- FAIL: FuzzFoo (9.05s) This is the same output it has while fuzzing, so if minimization runs for a long time (default allows 1 minute), then it looks like minimization is hanging. It's also confusing that the execs/sec would continually decrease. Now, when minimization is running, the command line output will look something like this: fuzz: minimizing 34995-byte crash input... fuzz: elapsed: 3s, minimizing fuzz: elapsed: 6s, minimizing fuzz: elapsed: 9s, minimizing fuzz: elapsed: 9s, minimizing --- FAIL: FuzzFoo (9.05s) The final "fuzz: elapsed: 6s, minimizing" could be printed twice because we always print one final log to the command line before we exit. Updates #48820 Change-Id: Ie5b9fde48b8d4e36e13a81ae50a6d69bf4d0dbe3 Reviewed-on: https://go-review.googlesource.com/c/go/+/354371 Trust: Katie Hockman <firstname.lastname@example.org> Run-TryBot: Katie Hockman <email@example.com> Reviewed-by: Jay Conrod <firstname.lastname@example.org>