aflsmart build is broken

[jonathanmetzman]

@thuanpv
Running make build-aflsmart-libpcap_fuzz_both gives me the following error:

Step 5/7 : RUN git clone https://github.com/aflsmart/aflsmart /afl &&     cd afl &&     git checkout df095901ea379f033d4d82345023de004f28b9a7 &&     AFL_NO_X86=1 make
fatal: reference is not a tree: df095901ea379f033d4d82345023de004f28b9a7
The command '/bin/sh -c git clone https://github.com/aflsmart/aflsmart /afl &&     cd afl &&     git checkout df095901ea379f033d4d82345023de004f28b9a7 &&     AFL_NO_X86=1 make' returned a non-zero code: 128
make: *** [docker/build.mk:193: .aflsmart-libpcap_fuzz_both-oss-fuzz-builder-intermediate] Error 128

I wonder why this wasn't caught by our CI. I will test this again to check.
What's also weird is how this affected the results of the last experiment: 2020-03-11
Some of the benchmarks are missing because it failed to build for them (e.g. bloaty_fuzz_target), but others such as freetype have data.

I'll point out that the aggregate ranking for aflsmart in that report is not really accurate because it is missing benchmarks.

[thuanpv]

Hi @jonathanmetzman. Thank you for your analysis of the issue. It seems to be flaky and it is very similar to this issue actions/checkout#23. The root cause could be that in the builder script I check out a merge commit. I am testing it locally with a normal commit to see how it goes.

[jonathanmetzman]

These are the builds that failed:
mbedtls_fuzz_dtlsclient
bloaty_fuzz_target
php_php-fuzz-parser
libpcap_fuzz_both
sqlite3_ossfuzz
openssl_x509
irssi_server-fuzz
zlib_zlib_uncompress_fuzzer
jsoncpp_jsoncpp_fuzzer
curl_curl_fuzzer_http
systemd_fuzz-link-parser

So I think the issue only affects oss-fuzz benchmarks, weird.

I also noticed afl++ failed on opnessl_x509

[jonathanmetzman]

Hi @jonathanmetzman. Thank you for your analysis of the issue. It seems to be very similar to this issue actions/checkout#23 and the root cause could be that in the builder script I check out a merge commit. I am testing it locally with a normal commit to see how it goes.

Cool thanks!

[thuanpv]

A quick update. If I use the newest commit (just run git clone with no git checkout command), even though it is also a merge commit, the issue is gone. So the issue is definitely due to the git checkout command but I don't know why. It is strange!

[jonathanmetzman]

I'll look into this, there may be something weird in our oss-fuzz builds.
Of course, to add to the confusion, the PR I put up failed.

[jonathanmetzman]

I may have made some progress by doing this clone/checkout towards the top of the Dockerfile (still need to verify this and figure out the root cause of course) but there seems to be another error after that. I'll first fix #115 which I think prevents this from being caught in CI.

[thuanpv]

As I said, ignoring "git checkout" helped me to fix the first issue (fatal: reference is not a tree). However, while compiling Peach which is a dependency of AFLSmart, it failed with the libpcap benchmark -- which is from Oss-Fuzz. The root cause is the C standard.

The following compilation command works for FuzzBench's libpng and libjpeg benchmarks but it failed for libpcap

CC=gcc-4.4 CXX=g++-4.4 CXXFLAGS="-std=c++0x" ./waf configure
CC=gcc-4.4 CXX=g++-4.4 CXXFLAGS="-std=c++0x" ./waf install

When I changed the commands to use "-std=gnu99" for both CXXFLAGS and CFLAGS, the configuration passed but the installation still failed.

I suspect the reason is that the libpcap benchmark, and possibly other benchmarks from Oss-Fuzz, uses a different base-builder image than the base-builder Dockerfile of FuzzBench and there is some conflicting settings/software versions between them.

Do you have any suggestions, @jonathanmetzman ?

Thanks

[jonathanmetzman]

I suspect the reason is that the libpcap benchmark, and possibly other benchmarks from Oss-Fuzz, uses a different base-builder image than the base-builder Dockerfile of FuzzBench and there is some conflicting settings/software versions between them.

Do you have any suggestions, @jonathanmetzman ?

They do in fact use a different base image. libpcap uses the builder image from OSS-Fuzz. Benchmarks This difference causes enough issues that I think this should be changed. Filed #125 to fix this.

In the meantime I think I figured out this issue (as well as the git one, which is a result of relying on CWD being '/' which is not the case in OSS-Fuzz). This issue was caused by the fact that OSS-Fuzz's builder image defines CFLAGS. Setting CFLAGS to "" fixed the issue, will update #110 (and land it before my CI fix lands). This issue is tough to solve in the general case because docker doesn't make it easy to delete env variables. We could set it to "" as we do here, but this affects builds of some fuzzers. For example I think AFL won't be built with -O3 if CFLAGS="".

[jonathanmetzman]

This issue is fixed.