fuzzer executions since May 26th

[caolanm]

Reviewing the LibreOffice fuzzer stats I see some big changes since May 26th.

https://oss-fuzz.com/v2/fuzzer-stats/by-day/2017-05-09/2017-06-09/fuzzer/libFuzzer_libreoffice_epsfuzzer
appears to still be still running but the view from May 26th onward is quite different with dramatically less tests_executed

while others like...

https://oss-fuzz.com/v2/fuzzer-stats/by-day/2017-05-09/2017-06-09/fuzzer/libFuzzer_libreoffice_wmffuzzer
seem to have fallen off a cliff since May 26 also

and

https://oss-fuzz.com/v2/fuzzer-stats/by-day/2017-05-09/2017-06-09/fuzzer/libFuzzer_libreoffice_ww8fuzzer
is a new fuzzer target since Jun 2nd and never seems to have had any tests_executed ?

is this just the allocation of available resources and things wax and wane over time. or is there something wrong ? If the issue is available resources could new targets within a project be prioritized over old targets ?

[Dor1s]

/cc @oliverchang

Hm, I see lack of libreoffice builds on May 28th and June 1st, 3rd, 4th, 5th, 6th, but not sure if that might cause troubles with stats either with jobs scheduling.

Also, there were some changes related to jobs / stats recently, but it doesn't seem to be a cause, because other fuzzers (I checked zlib and one fuzzer from boringssl) have proper stats for the same dates.

If the issue is available resources could new targets within a project be prioritized over old targets ?

No, targets in a project have the same priority as of now.

[oliverchang]

Were there any recent changes wrt to signal handling in the targets? (in particular, SIGALRM).

Looking at the logs, it looks like targets are exiting very early without any output.

I can reproduce this with e.g. epsfuzzer with -timeout=1.

Putting a breakpoint on _exit shows:

#0  __interceptor__exit (status=255) at /src/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:3741
#1  0x00000000016807fa in (anonymous namespace)::callSystemHandler(int, siginfo_t*, void*) () at /src/libreoffice/sal/osl/unx/signal.cxx:367
#2  0x000000000167fd4f in (anonymous namespace)::signalHandlerFunction(int, siginfo_t*, void*) () at /src/libreoffice/sal/osl/unx/signal.cxx:489
#3  <signal handler called>
#4  ForEachNonZeroByte<(lambda at /src/libfuzzer/FuzzerTracePC.h:135:28)> () at /src/libfuzzer/FuzzerTracePC.h:123
#5  CollectFeatures<(lambda at /src/libfuzzer/FuzzerLoop.cpp:400:23)>(void) () at /src/libfuzzer/FuzzerTracePC.h:148
#6  0x0000000002090a36 in RunOne () at /src/libfuzzer/FuzzerLoop.cpp:400
#7  0x0000000002092312 in MutateAndTestOne () at /src/libfuzzer/FuzzerLoop.cpp:596
#8  0x0000000002092ad8 in Loop () at /src/libfuzzer/FuzzerLoop.cpp:624
#9  0x0000000002075159 in FuzzerDriver () at /src/libfuzzer/FuzzerDriver.cpp:744
#10 0x0000000002068b39 in main () at /src/libfuzzer/FuzzerMain.cpp:20

@kcc I think libFuzzer installs its own SIGALRM handler to detect timeouts, but libreoffice is overriding this?

[oliverchang]

(if you're installing a handler for SIGSEGV/SIGABRT too, then we likely won't be able to report those properly either since we won't get an sanitizer stacktrace).

[kcc]

Ouch. yea.
libFuzzer installs its own ALRM handler and it also sets an alarm.
If the users redefined ALRM handler, we are screwed.

1. need to remove openoffice's ALRM handler
2. need to figure out how to detect this case in future.

[caolanm]

I can remove the SIGALRM handler and see if that makes a difference, but I'm fairly sure that has always been in there since before this problem began

[oliverchang]

It could also be that a very easy to reach timeout (or some other bug that invokes signal handler) was recently introduced, causing targets to exit very quickly and thus result in the lower number of executions. We might have just gotten lucky before.

[caolanm]

Well, that worked. Numbers are back up to the 1/3 billion executions a day per target and results are flowing again