How long it takes to finish fuzzing?

[RazrFalcon]

It's not directly related to the cargo-fuzz, more to the llvm.

I'm new to fuzz-testing and can't understand how long it takes to finish fuzzing. Is it endless? Is there a progress somewhere in the output?

Now I'm just shutting down it after ~10 minutes.

[0xcpu]

NOTE: This is a very simple explanation. I'm not a fuzzing expert

It can be so that you fuzz for days and don't find a crash. It depends on some properties, like:

• How and what are you fuzzing;
• What fuzzer you're using;
  There are different types of fuzzers
• How much resources you have;
  Time, computing resources

You can monitor the coverage of libfuzzer to observe the progress of the fuzzing process.

Hope this helps a bit.

[RazrFalcon]

Thanks.

monitor the coverage

What exactly do this numbers mean?

[0xcpu]

cov: Total number of code blocks or edges covered by the executing the current corpus.

Reference: http://llvm.org/docs/LibFuzzer.html#output

[Manishearth]

Fuzzing is endless; it will continue to run forever. You're supposed to leave it running for a long time in the background to see if it returns anything.

[RazrFalcon]

Thanks.

[emk]

I think it's worth understanding:

1. How to tell when a fuzzing run is making progress, and
2. How to tell when it's just burning cycles with very low chance of reward.

My first couple attempts with cargo fuzz stalled out very quickly because of bugs in my harness. Once I figured out how to read the output and tell when things were working, it was a lot easier to get good results.

@RazrFalcon: After looking at the link provided by @ner0x652, you might also be interested in reading through the libfuzzer tutorial, which talks a bit about the workflow for C programs.

Here's an example of what things look like when they're working:

Loading corpus dir: corpus
#0	READ units: 105
#105	INITED cov: 1581 corp: 76/9695b exec/s: 0 rss: 126Mb

For this run, I started with 105 files in fuzz/corpus. These represent "interesting" possible inputs for the fuzzer to mutate. Some of these were generated by hand, and others were generated by previous runs. The cov: 1581 says that all the test cases taken together manage to reach 1581 "basic blocks" (straight-line sections of code with between branches). The corp: 76/... indicates that my 105-file corpus was reduced down to 76 inputs that reach all known blocks.

We quickly discover two new inputs:

#543	NEW    cov: 1582 corp: 77/9881b exec/s: 0 rss: 134Mb L: 186 MS: 3 ChangeBinInt-InsertByte-InsertRepeatedBytes-
#5158	NEW    cov: 1583 corp: 78/10057b exec/s: 0 rss: 183Mb L: 176 MS: 3 ShuffleBytes-ChangeBinInt-ShuffleBytes-

We can now access 1583 basic blocks using 78 files. The new files are written to fuzz/corpus so we can reuse them for the next run.

And then a third new input:

#15775	NEW    cov: 1584 corp: 79/10254b exec/s: 7887 rss: 242Mb L: 197 MS: 5 InsertRepeatedBytes-ShuffleBytes-ManualDict-ChangeByte-CMP- DE: "\x06"-"\x00\x00\x01\xba"-

Here, we can also see exec/s: 7887, which means that we're testing 7,887 inputs/second, which is excellent. (Anything over 1,000/second is pretty good, although I found five panics with just 250/second.)

And now things start to slow down:

#16384	pulse  cov: 1584 corp: 79/10254b exec/s: 8192 rss: 245Mb
#32768	pulse  cov: 1584 corp: 79/10254b exec/s: 6553 rss: 327Mb
#65536	pulse  cov: 1584 corp: 79/10254b exec/s: 5957 rss: 505Mb

We've tried 65,536 inputs without seeing any improvements in coverage. There may still be bugs, but they're not easy to reach.

And now let's leave it running all afternoon:

#134217728	pulse  cov: 1584 corp: 79/10254b exec/s: 6868 rss: 646Mb

We're now at 130 million inputs, and we're still stuck at cov: 1584 ever since input 15,775. (And I've started parallel fuzzers that have tried another 300+ million inputs, and come up dry.)

At this point, it's safe to say that the fuzzer has stalled: It has pretty good coverage, and it isn't finding any way to increase coverage despite trying close to half billion inputs. Now, if I left it running all week, I might find another bug. But my time is probably better spent inventing new inputs for fuzz/corpus. So I'm happy hitting Control-C at this point and letting my laptop fan take a break.

(It would be nice to have something along the lines of #67, which would allow us to know which functions and lines of code haven't been reached yet. Then I could create new examples by hand to reach hard-to-reach corners of the code.)

[RazrFalcon]

Thanks for details!