Improve handling of non-reproducible crashes

[saelo]

As with any fuzzer, Fuzzilli can trigger crashes that do not reproduce later on. There can be various reasons for that, e.g. memory pressure of the system during the time of the crash or simply non-deterministic behaviour of the target. As JavaScript engines are quite complex, it is expected that there will be non-reproducible crashes during fuzzing.

It would, however, be nice to have some more information about how a sample originally crashed so root-causing of non-reproducible crashes becomes easier/possible. This could e.g. be done by adding a commandline flag that, when enabled, causes Fuzzilli to record stdout + stderr of the target during fuzzing and include that in the crashing testcases. That way, it should at least be somewhat clear how the sample originally crashed.

There could also be a bug in Fuzzilli that causes it to incorrectly treat some files as crashes under some circumstances. One approach to debug this would be to comment out this line:

fuzzilli/Sources/libreprl/libreprl.c

Line 106 in 2e7c967

dup2(devnull, 2);

and thus log stderr of the target during fuzzing. That should show whether the target actually crashed and roughly how it crashed (e.g. with an assertion failure).

[saelo]

One way to deal with stderr/stdout output for crashes would be to add a new Operations "Comment" which contains an arbitrary string as payload and is then lifted to a JS comment. Then, the Fuzzer.execute method could, for crashes, append a comment instruction to the crashing program with the content of stderr and/or stdout. This way, the output would automatically be preserved even when sending the crashing program to a master instance.

[saelo]

One way to deal with stderr/stdout output for crashes would be to add a new Operations "Comment" which contains an arbitrary string as payload and is then lifted to a JS comment. Then, the Fuzzer.execute method could, for crashes, append a comment instruction to the crashing program with the content of stderr and/or stdout. This way, the output would automatically be preserved even when sending the crashing program to a master instance.

Done: a24e205

[saelo]

Closing this issue for now to see if that is enough