Feature request: timeout -t

[floyd-fuh]

Would it be possible to get the -t switch to specify a timeout for a testcase? Fuzzing is only efficient when the inputs do not take too long. I'm currently trying to do another run with Tika, but alredy the fourth input dry run is just stuck for several minutes...

[floyd-fuh]

As you said in another issue:

JQF does not handle hangs at all at this point, which is a limitation (mainly because we are interested in unit tests and not long-running programs). I'm trying to find good ways to kill running threads preemptively without breaking too much (Thread.stop in the JDK is deprecated, and Future.cancel or Thread.interrupt leaks resources because the threads aren't actually killed). If you have any suggestions here, I would love to hear them!

My first suggestion is to try implementing Thread.interrupt as suggested in the docs, for some cases that might help. I would implement this as a first attempt to stop the thread.

My second suggestion is maybe a little uncommon, but I would say let's just ignore the deprecation and call stop(). While not very nice, this is fuzzing-land where we simply need this kind of control, because AFL fuzzing will always find a way to hang Java. As the same doc link suggest, it's probably best to catch ThreadDeath, even if that's not optimal.

There are also suggestions to use the new JDK 9 spinWait(), but I'm not sure that method would help.

A different approach would be using ExecutorService. This is also how Kelinci does it or also explained on stackoverflow.

[rohanpadhye]

I've added some limited support for the -t option when using jqf-afl-fuzz. The only snag is that some timeouts may get misclassfied as "crashes" instead of "hangs". This is because I'm trying to stick to using stock AFL instead of having a custom AFL for JQF, which means I have to fool AFL into believing that it has killed the target program on timeout by mocking signals. In the worst case, the signal will appear before AFL's timer goes off and it will therefore be classified as a crash instead of a hang.

Note: You will have to re-run the make command in addition to mvn package (or simply setup.sh which does both) because the AFL proxy has also been modified.

The default value for timeout is 10000 (milliseconds) to prevent spurious timeouts, but I've tried fuzzing Tika with 5000 on my machine and it was fine.

The implementation does not use either Thread.stop() nor Thread.interrupt(). Instead, since we are instrumenting program branches for fuzzing, we can simply monitor how much time has elapsed after every few thousand branches (without incurring too much overhead), and then throw a run-time exception if the elapsed time is more than the set timeout. This idea was suggested by Koushik Sen.

This trick means that JQF can detect hangs due to infinite loops fairly efficiently, but not hangs due to deadlocks where no branches are being executed. For now, I will say this is sufficient as most fuzzing targets are not I/O intensive. In the future, we can possible interrupt deadlocks using Thread.interrupt().

[rohanpadhye]

This is still not working as expected. Although the first hang gets detected, throwing a TimeoutException from within an instrumentation handler seems to corrupt the instrumentation tracking itself and subsequent hangs do not get detected.

I'm still working on it... the issue remains open.

[floyd-fuh]

I just removed the -t argument for one of my runs, and while it seems to be running fine it says in the process list /usr/local/bin/afl-fuzz -d -i ./tika-corpus-only-small -o tika-out2 -T edu.berkeley.cs.jqf.examples.tika.TikaParserTest#fuzz -m 2000 -t 50 /opt/jqf/bin/jqf-afl-target -v edu.berkeley.cs.jqf.examples.tika.TikaParserTest fuzz @@. Is that correct that it uses a timeout of 50ms now? Or is that something I shouldn't be concerned about?

[rohanpadhye]

@floyd-fuh Don't worry about that. We always give AFL a 50ms timeout regardless of what the actual timeout is. This is so that AFL sets its timeout flag for every single input. In practice, AFL only marks a run as a "hang" if the timeout flag is set AND the return value of waitpid() on the forked fuzz target indicates SIGKILL. In JQF, we always let AFL's timeout flag fire (in 50ms) and handle the real timing measurements in Java. When a TimeoutException is thrown, we lie to AFL and tell it that the process ended with SIGKILL.

[floyd-fuh]

Ah, perfect, thanks for the pointers! I think I just thought it wouldn't work correctly, but everything is fine.

Btw. I ran into new problems with the timeouts. JQF classified the Java bug found in #27 as a crash. As you described above not what you intended, but perfectly fine in my use case. So a miss-classification as crash or hang is fine for me. However, finding a similar bug in a Java library JQF actually got stuck. I think the reason is that the endless loop is in a part of the code that was not instrumented, so JQF won't be able to measure time and throw a run-time exception after the elapsed time.

So at the moment I have to recheck on my fuzzers from time to time. Today they were running for 5 days, but got stuck after the first 7 hours. I wasn't too unhappy because that meant a new finding, but it's obviously not optimal having to check on them and restart fuzzers often.

So I guess a Thread.interrupt solution would be really helpful.

And sorry for asking for so many features, I kind of feel bad, but JQF is really improving a lot and I clearly prefer it over Kelinci and java-afl at the moment. Keep up the good work 👍

[rohanpadhye]

Timeouts are working fine in AFL-mode for now. Please re-open if issues persist.

[floyd-fuh]

I updated the version of JQF I'm using to the latest github version and started fuzzing Apache Tika again. However, as you said AFL timeout should work fine now, I ran JQF with lower timeouts, such as 1000 or even 300. It runs fine and also the execs/s seem to show that it is going faster. However, stock AFL would usually put a lot of files in the "hangs" directory and would very quickly find hangs. However, JQF never puts any file in the "hangs" directory for me now, which is very weird with these timeout settings.

So when I run with -t 300 I expect every file that takes more than 300ms of processing to be put in the hangs directory. This doesn't seem to be the case. So my guess is at the moment JQF is not able to find endless loops as it doesn't fill the hangs directory. Can you confirm this?

Can you elaborate a little on how the timeout setting is applied?