libreoffice failing check_build in test_all with simultaneous checks

[caolanm]

The libreoffice builds have been flagged as broken for quite a while with crashes in the afl++ bad_build checks with the typical "Whoops, the target binary crashed suddenly, before receiving any input from the fuzzer!" error.

I can reproduce locally testing the full set of of ~45 fuzzers with
python infra/helper.py check_build --sanitizer address --engine afl --architecture x86_64 libreoffice
but on testing just one at a time each passes ok. I have determined that the problem seems to be triggered by running multiple fuzzers in parallel by test_all in infra/base-images/base-runner/test_all.py

If I (on an 8core 32G) limit running them in parallel by changing...

pool = multiprocessing.Pool()

to

processes = max(1, os.cpu_count() // 4) # so just 2 for me locally
pool = multiprocessing.Pool(processes)

then all the bad_build checks pass successfully, with 4 processes there tends to be some failures, and more failures the larger I allow the parallelization

[inferno-chromium]

you want to propose a PR for this ?

[jonathanmetzman]

you want to propose a PR for this ?

I'm not sure what the fix is. Changing this to sequential execution might slow things down quite a bit. We may want to do something hacky instead, like retry failed cases in sequence.

[vanhauser-thc]

I made a few fixes to afl++, it builds the fuzzers fine now (for a few days), however every input times out., even with a timeout of 60 seconds per input, and the input being just a linefeed.
when called as ./fuzz-target corpus/* then it behaves fine.
This only happens when the fuzz-target is run in afl-fuzz.
I used gcore on the spawned off target and it wastes its time in an libreoffice wait... function.

I am pretty clueless what is happening and why it is happening only with afl++ and only when run via afl-fuzz.
I can assist debugging this, but I think as a first step this needs someone with libreoffice debugging knowledge to see where and why it hangs inside libreoffice functions.

EDIT: hmm that it builds and runs but inputs only timing out is what is happening on my system. in the oss-fuzz build system logs I see it still "crashing" (which basically means for libreoffice that something in LLVMFuzzerInitialize is failing and hence an abort() is triggered?)

[caolanm]

debugging what I see a little further I see that in the high-load many process launched case
a) in src/afl-forkserver.c afl_fsrv_start the read_s32_timed call returns 0 and that triggers kill(fsrv->fsrv_pid, fsrv->kill_signal); (SIGKILL)
b) read_s32_timed returns 0 because *stop_soon_p is non-zero at restart_read:
c) *stop_soon_p becomes non-zero in handle_stop_sig of src/afl-fuzz-init.c due to receiving SIGINT
d) that SIGINT is sent by the timeout calls in bad_build_check so it is that "outer" timeout process which is sending SIGINT which then triggers afl-forkserver's internal SIGKILL

[caolanm]

wrt...

... every input times out., even with a timeout of 60 seconds per input, and the input being just a linefeed.
when called as ./fuzz-target corpus/* then it behaves fine.
This only happens when the fuzz-target is run in afl-fuzz.

Within oss-fuzz infra/base-images/base-runner/bad_build_check has AFL_DRIVER_DONT_DEFER=1 which is the magic smoke to make https://raw.githubusercontent.com/llvm/llvm-project/main/compiler-rt/lib/fuzzer/afl/afl_driver.cpp not use the default persistent_mode/deferred_fork_server so the lack of export AFL_DRIVER_DONT_DEFER=1 is presumably why the fuzzers get stuck in wait in a local test outside of bad_build_check

[vanhauser-thc]

I am trying now for three days to get this build but always run into:

 ---> Running in 7b4b398ef5bd
Cloning into 'libreoffice'...
fatal: read error: Connection timed out
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
The command '/bin/sh -c git clone --depth 1 git://anongit.freedesktop.org/libreoffice/core libreoffice' returned a non-zero code: 128

so I have difficulties helping here :(

debugging what I see a little further I see that in the high-load many process launched case
a) in src/afl-forkserver.c afl_fsrv_start the read_s32_timed call returns 0 and that triggers kill(fsrv->fsrv_pid, fsrv->kill_signal); (SIGKILL)
b) read_s32_timed returns 0 because *stop_soon_p is non-zero at restart_read:
c) *stop_soon_p becomes non-zero in handle_stop_sig of src/afl-fuzz-init.c due to receiving SIGINT
d) that SIGINT is sent by the timeout calls in bad_build_check so it is that "outer" timeout process which is sending SIGINT which then triggers afl-forkserver's internal SIGKILL

that looks like a sound explanation what is happening in the build test.
So the solution would be not to run them all at one, but one after another?

... every input times out., even with a timeout of 60 seconds per input, and the input being just a linefeed.
when called as ./fuzz-target corpus/* then it behaves fine.
This only happens when the fuzz-target is run in afl-fuzz.

Within oss-fuzz infra/base-images/base-runner/bad_build_check has AFL_DRIVER_DONT_DEFER=1 which is the magic smoke to make https://raw.githubusercontent.com/llvm/llvm-project/main/compiler-rt/lib/fuzzer/afl/afl_driver.cpp not use the default persistent_mode/deferred_fork_server so the lack of export AFL_DRIVER_DONT_DEFER=1 is presumably why the fuzzers get stuck in wait in a local test outside of bad_build_check

the problem that I describe is outside of the build test.
When I just build that fuzz target and then run any of the fuzz targets within the docker container manually, e.g.:

cd /out
mkdir in
cd in
unzip ../xlsxfuzzer_seed_corpus.zip
cd ..
afl-fuzz -i in -o out -t 10000+ -- ./xlsxfuzzer

every single test case times out. I sent killall -SIGSTOP afl-fuzz xlsxfuzzer and then used gcore on the active xslxfuzzer process and analyzed the core. it was hanging in an internal libreoffice_wait... function.

[caolanm]

I am trying now for three days to get this build but always run into:
The command '/bin/sh -c git clone --depth 1 git://anongit.freedesktop.org/libreoffice/core libreoffice' returned a non-zero code: 128

There has been some connectivity issues at anongit.freedesktop.org recently, though it seems ok again since the last few hours. If it returns as a problem git://git.libreoffice.org/core is an alternative url

the problem that I describe is outside of the build test.
When I just build that fuzz target and then run any of the fuzz targets within the docker container manually, e.g.:

cd /out
mkdir in
cd in
unzip ../xlsxfuzzer_seed_corpus.zip
cd ..
afl-fuzz -i in -o out -t 10000+ -- ./xlsxfuzzer

every single test case times out. I sent killall -SIGSTOP afl-fuzz xlsxfuzzer and then used gcore on the active xslxfuzzer process and analyzed the core. it was hanging in an internal libreoffice_wait... function.

yes, this sounds like a missing AFL_DRIVER_DONT_DEFER=1 in your environment (where #1449 has some backstory on that). Seeing as this oddity causes confusion I've pushed https://cgit.freedesktop.org/libreoffice/core/commit/?id=5f762b34bb1f93aeb409060d74b8e38ab75a8732 which seems to do away with the need for AFL_DRIVER_DONT_DEFER

d) that SIGINT is sent by the timeout calls in bad_build_check so it is that "outer" timeout process which is sending SIGINT which then triggers afl-forkserver's internal SIGKILL

that looks like a sound explanation what is happening in the build test.
So the solution would be not to run them all at one, but one after another?

possibly, I wondered if instead of using timeout for wallclock time and instead ulimit -t for cpu time was worth exploring, but I think I'll try instead the suggestion of retrying the parallel failures serially

[vanhauser-thc]

the problem that I describe is outside of the build test.
When I just build that fuzz target and then run any of the fuzz targets within the docker container manually, e.g.:

cd /out
mkdir in
cd in
unzip ../xlsxfuzzer_seed_corpus.zip
cd ..
afl-fuzz -i in -o out -t 10000+ -- ./xlsxfuzzer

every single test case times out. I sent killall -SIGSTOP afl-fuzz xlsxfuzzer and then used gcore on the active xslxfuzzer process and analyzed the core. it was hanging in an internal libreoffice_wait... function.

yes, this sounds like a missing AFL_DRIVER_DONT_DEFER=1 in your environment (where #1449 has some backstory on that). Seeing as this oddity causes confusion I've pushed https://cgit.freedesktop.org/libreoffice/core/commit/?id=5f762b34bb1f93aeb409060d74b8e38ab75a8732 which seems to do away with the need for AFL_DRIVER_DONT_DEFER

OK maybe this is a misunderstanding.
In normal oss-fuzz usage there is no AFL_DRIVER_DONT_DEFER set - unless the fuzzing target is unable to run otherwise, and then should be clearly documented.
Plus I am not even sure that oss-fuzz and clusterfuzz do even support this, as this would be needed to be set in the fuzzing environment. And per project fuzzing settings (environment variable, command options, etc.) are not supported AFAIK.

[vanhauser-thc]

@caolanm @inferno-chromium
libreoffice fails now with:

TIMEOUT
ERROR: context deadline exceeded
Step #26: <?xml version='1.0' encoding='UTF-8'?><Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Details>Request signature expired at: 2021-04-07T18:08:34+00:00</Details></Error>

looks like this comes from the build systems because the build takes too long?

[caolanm]

@caolanm @inferno-chromium
looks like this comes from the build systems because the build takes too long?

I think so, but it can't be related to this change seeing as its a timeout long before an attempt to run the fuzzers, so just bad luck I think that a new issue has arisen which blocks seeing if this older issue is resolved.

The timeout may be quite close to the length of time it would take to complete successfully as everything was compiled by the timeout on the 7th April and about a third of the fuzzers were linked when it timeouted. It is possible that #5588 might squeak the build in under the timeout, though that is also running into a different set of difficulties with the config used to do per-integration builds

[vanhauser-thc]

I didnt meant to imply this PR is the reason :)
just following up to see if everything works now and if I can help

[caolanm]

https://oss-fuzz-build-logs.storage.googleapis.com/index.html#libreoffice we have a successful build again, excellent.