[aflplusplus] Fix target compilation, we had 10%+ speed loss ... (#377)

* Remove dislocator from AFL++ One of the possible reasons of the decrement in performance of AFL++ in the lastest report is due to the missing AFL_ALIGNED_ALLOC env var. Without this variable, dislocator's malloc does not ensure that the returned address is aligned to max_aligned_t (as required by posix) but it is able, in this way, to catch more bugs. Many applications checks for this alignment and this may cause an early exit in the application (and so a low coverage). I completely remove dislocator to better understand what is happening in the next run of the experiments. * update AFL++ to fix #110 * afl++ cmplog * fix afl++ schedules flags error * unification of aflplusplus confs * update fuzzers.yml * make presubmit happy * add marc changes from #229 * presubmit * add also ngram2/4 * runner for cmplog * update commit * prepare new features * new variants * add support for ngram5 * add ctx variants * format * less lint more fire * seriously? one instead of two spaces is an issue? * add lto variant * update checkout * fix build lines * formatting * libfuzzer valueprofiles variant * formatting * formatting * remove old afl++ variants * update git * add afl++ variants * final afl++ variants * remove valuprofile and old symlink * fix laf-intel crash * update commit * fix for older llvm versions * fix bug in laf-intel for sqlite3 testcase * switch afl++ variants * remove LTO * fix fuckup in main fuzzer.py * update git * LTO submit * lto update * copy llvm11 c++ libs to targets * fix LTO and add 2nd LTO variant * remove 2 fuzzer variants to make space for LTO * LTO compiles curl target now, ensure flags are empty * fix tcpdump benchmark * jsoncpp needs a dynamic map too :-( * build and run all benchmarks * switch to tracepcguard * Revert "switch to tracepcguard" This reverts commit 782f555. * switch from instrim to tracepcguard * fix a fuzzer * fix formatting * remove temporary fuzzers * fix generate report document * fix report document, add report alternative * remove report changes * remove report changes * next variants * Revert "next variants" This reverts commit c551d90. * add new variants * nitpicks * remove fuzzer variants * remove variants from fuzzer.yaml * add optimal variant * formatting * fix builds * more build fixes * more build fixes * more build fixes * more build fixes * fix builds * various fixes and enhancements * formatting * formatting * fix. too tired * fix. too tired * add splitting floats to laf * enable lto for the relevant targets again after fixing bugs in laf+cmplog * freetype2 needs dynamic map * fix target compilation * fix target compilation * final touches for now * done performance tuning * add shmem variant Co-authored-by: Andrea Fioraldi <andreafioraldi@gmail.com> Co-authored-by: root <root@localhost.localdomain>
google · May 26, 2020 · 64c96ee · 64c96ee
1 parent 7844c67
commit 64c96ee
Show file tree

Hide file tree

Showing 7 changed files with 243 additions and 14 deletions.
diff --git a/.github/workflows/fuzzers.yml b/.github/workflows/fuzzers.yml
@@ -18,6 +18,7 @@ jobs:
           - aflfast
           - aflplusplus
           - aflplusplus_optimal
+          - aflplusplus_shmem
           - aflsmart
           - eclipser
           - entropic

diff --git a/fuzzers/aflplusplus/builder.Dockerfile b/fuzzers/aflplusplus/builder.Dockerfile
@@ -24,11 +24,12 @@ RUN apt-get update && \
 # Set AFL_NO_X86 to skip flaky tests.
 RUN git clone https://github.com/AFLplusplus/AFLplusplus.git /afl && \
     cd /afl && git checkout dev && \
-    git checkout fc574086ec8beff72a032f73884fb9f1f0d02f47 && \
-    AFL_NO_X86=1 CFLAGS= CXXFLAGS= make PYTHON_INCLUDE=/ && \
-    cd llvm_mode && CFLAGS= CXXFLAGS= make
+    git checkout 996e1515b320fb2d44c367dea7b4d26f2d56f5df && \
+    unset CFLAGS && unset CXXFLAGS && \
+    AFL_NO_X86=1 CC=clang PYTHON_INCLUDE=/ make && \
+    cd llvm_mode && make
 
 # Use afl_driver.cpp from LLVM as our fuzzing library.
 RUN wget https://raw.githubusercontent.com/llvm/llvm-project/5feb80e748924606531ba28c97fe65145c65372e/compiler-rt/lib/fuzzer/afl/afl_driver.cpp -O /afl/afl_driver.cpp && \
-    clang++ -stdlib=libc++ -std=c++11 -O2 -c /afl/afl_driver.cpp && \
-    ar ru /libAFLDriver.a *.o
+    clang++ -O3 -funroll-loops -stdlib=libc++ -std=c++11 -c /afl/afl_driver.cpp && \
+    ar ru /libAFLDriver.a afl_driver.o
diff --git a/fuzzers/aflplusplus_optimal/builder.Dockerfile b/fuzzers/aflplusplus_optimal/builder.Dockerfile
@@ -25,22 +25,26 @@ RUN apt-get update && \
     wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - && \
     apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 1E9377A2BA9EF27F && \
     apt-get update && \
-    apt-get install -y clang-11 clang-11-doc clang-11-examples clangd-11 clang-format-11 clang-tidy-11 clang-tools-11 libc++1-11 libc++-11-dev libc++abi1-11 libc++abi-11-dev libclang1-11 libclang-11-dev libclang-common-11-dev libclang-cpp11 libclang-cpp11-dev libfuzzer-11-dev liblld-11 liblld-11-dev liblldb-11 liblldb-11-dev libllvm11 libomp-11-dev libomp-11-doc libomp5-11 lld-11 lldb-11 llvm-11 llvm-11-dev llvm-11-runtime llvm-11-tools && \
+    apt-get install -y clang-11 clangd-11 clang-tools-11 libc++1-11 libc++-11-dev \
+      libc++abi1-11 libc++abi-11-dev libclang1-11 libclang-11-dev libclang-common-11-dev \
+      libclang-cpp11 libclang-cpp11-dev liblld-11 liblld-11-dev liblldb-11 \
+      liblldb-11-dev libllvm11 libomp-11-dev libomp5-11 lld-11 lldb-11 \
+      llvm-11 llvm-11-dev llvm-11-runtime llvm-11-tools && \
     apt-get install -y gcc-9 g++-9
 
 # Download and compile afl++ (v2.62d).
 # Build without Python support as we don't need it.
 # Set AFL_NO_X86 to skip flaky tests.
 RUN git clone https://github.com/AFLplusplus/AFLplusplus.git /afl && \
     cd /afl && \
-    git checkout fc574086ec8beff72a032f73884fb9f1f0d02f47 && \
-    AFL_NO_X86=1 CFLAGS= CXXFLAGS= make PYTHON_INCLUDE=/ && \
-    export LLVM_CONFIG=llvm-config-11 && \
-    cd llvm_mode && CFLAGS= CXXFLAGS= REAL_CC=gcc-9 REAL_CXX=g++-9 make && \
+    git checkout 996e1515b320fb2d44c367dea7b4d26f2d56f5df && \
+    unset CFLAGS && unset CXXFLAGS && \
+    AFL_NO_X86=1 CC=clang PYTHON_INCLUDE=/ make && \
+    cd llvm_mode && LLVM_CONFIG=llvm-config-11 REAL_CC=gcc-9 REAL_CXX=g++-9 make && \
     make install
 
 # Use afl_driver.cpp from LLVM as our fuzzing library.
 RUN wget https://raw.githubusercontent.com/llvm/llvm-project/5feb80e748924606531ba28c97fe65145c65372e/compiler-rt/lib/fuzzer/afl/afl_driver.cpp -O /afl/afl_driver.cpp && \
-    clang++ -stdlib=libc++ -std=c++11 -O2 -c /afl/afl_driver.cpp && \
-    ar ru /libAFLDriver.a *.o && \
-    cp -a `llvm-config-11 --libdir`/libc++* /afl/
+    clang++ -O3 -funroll-loops -stdlib=libc++ -std=c++11 -c /afl/afl_driver.cpp && \
+    ar ru /libAFLDriver.a afl_driver.o && \
+    cp -va `llvm-config-11 --libdir`/libc++* /afl/
diff --git a/fuzzers/aflplusplus_optimal/fuzzer.py b/fuzzers/aflplusplus_optimal/fuzzer.py
@@ -44,7 +44,7 @@ def build():  # pylint: disable=too-many-branches,too-many-statements
     elif benchmark_name == 'lcms-2017-03-21':
         aflplusplus_fuzzer.build("lto", "nozero", "cmplog")
     elif benchmark_name == 'libjpeg-turbo-07-2017':
-        aflplusplus_fuzzer.build("tracepc", "nozero", "laf")
+        aflplusplus_fuzzer.build("lto", "nozero", "laf")
     elif benchmark_name == 'libpcap_fuzz_both':
         aflplusplus_fuzzer.build("lto", "nozero", "laf")
     elif benchmark_name == 'libpng-1.2.56':

diff --git a/fuzzers/aflplusplus_shmem/builder.Dockerfile b/fuzzers/aflplusplus_shmem/builder.Dockerfile
@@ -0,0 +1,32 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG parent_image=gcr.io/fuzzbench/base-builder
+FROM $parent_image
+
+# Install wget to download afl_driver.cpp. Install libstdc++ to use llvm_mode.
+RUN apt-get update && \
+    apt-get install wget libstdc++-5-dev -y
+
+# Download and compile afl++ (v2.62d).
+# Build without Python support as we don't need it.
+# Set AFL_NO_X86 to skip flaky tests.
+RUN git clone https://github.com/AFLplusplus/AFLplusplus.git /afl && \
+    cd /afl && git checkout dev && \
+    git checkout 1cae68dde32abf9c7fe83cb9a91890deba973834 && \
+    unset CFLAGS && unset CXXFLAGS && \
+    AFL_NO_X86=1 CC=clang PYTHON_INCLUDE=/ make && \
+    make -C llvm_mode && \
+    make -C examples/aflpp_driver && \
+    cp examples/aflpp_driver/libAFLDriver.a /
diff --git a/fuzzers/aflplusplus_shmem/fuzzer.py b/fuzzers/aflplusplus_shmem/fuzzer.py
@@ -0,0 +1,176 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Integration code for AFLplusplus fuzzer."""
+
+import os
+import shutil
+
+from fuzzers.afl import fuzzer as afl_fuzzer
+from fuzzers import utils
+
+# OUT environment variable is the location of build directory (default is /out).
+
+
+def get_cmplog_build_directory(target_directory):
+    """Return path to CmpLog target directory."""
+    return os.path.join(target_directory, 'cmplog')
+
+
+def build(*args):  # pylint: disable=too-many-branches,too-many-statements
+    """Build benchmark."""
+    # BUILD_MODES is not already supported by fuzzbench, meanwhile we provide
+    # a default configuration.
+    build_modes = list(args)
+    if 'BUILD_MODES' in os.environ:
+        build_modes = os.environ['BUILD_MODES'].split(',')
+
+    build_directory = os.environ['OUT']
+
+    # If nothing was set this is the default:
+    if not build_modes:
+        build_modes = ['tracepc', 'nozero']
+
+    # Instrumentation coverage modes:
+    if 'lto' in build_modes:
+        os.environ['CC'] = '/afl/afl-clang-lto'
+        os.environ['CXX'] = '/afl/afl-clang-lto++'
+        os.environ['RANLIB'] = 'llvm-ranlib-11'
+        os.environ['AR'] = 'llvm-ar-11'
+    elif 'qemu' in build_modes:
+        os.environ['CC'] = 'clang'
+        os.environ['CXX'] = 'clang++'
+    else:
+        os.environ['CC'] = '/afl/afl-clang-fast'
+        os.environ['CXX'] = '/afl/afl-clang-fast++'
+
+    if 'instrim' in build_modes:
+        # We dont set AFL_LLVM_INSTRIM_LOOPHEAD for better coverage
+        os.environ['AFL_LLVM_INSTRIM'] = 'CFG'
+    elif 'tracepc' in build_modes:
+        os.environ['AFL_LLVM_USE_TRACE_PC'] = '1'
+    elif 'classic' in build_modes:
+        os.environ['AFL_LLVM_INSTRUMENT'] = 'CLASSIC'
+
+    # Instrumentation coverage options:
+    # Do not use a fixed map location (LTO only)
+    if 'dynamic' in build_modes:
+        os.environ['AFL_LLVM_MAP_DYNAMIC'] = '1'
+    # Skip over single block functions
+    if 'skipsingle' in build_modes:
+        os.environ['AFL_LLVM_SKIPSINGLEBLOCK'] = '1'
+    # Enable context sentitivity for LLVM mode (non LTO only)
+    if 'ctx' in build_modes:
+        os.environ['AFL_LLVM_CTX'] = '1'
+    # Enable N-gram coverage for LLVM mode (non LTO only)
+    if 'ngram2' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '2'
+    elif 'ngram3' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '3'
+    elif 'ngram4' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '4'
+    elif 'ngram5' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '5'
+    elif 'ngram6' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '6'
+    elif 'ngram7' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '7'
+    elif 'ngram8' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '8'
+    elif 'ngram16' in build_modes:
+        os.environ['AFL_LLVM_NGRAM_SIZE'] = '16'
+
+    # Further instrumentation options:
+    # Disable neverZero implementation
+    if 'nozero' in build_modes:
+        os.environ['AFL_LLVM_SKIP_NEVERZERO'] = '1'
+
+    # Only one of the following OR cmplog
+    # enable laf-intel compare splitting
+    if 'laf' in build_modes:
+        os.environ['AFL_LLVM_LAF_SPLIT_SWITCHES'] = '1'
+        os.environ['AFL_LLVM_LAF_SPLIT_COMPARES'] = '1'
+        os.environ['AFL_LLVM_LAF_SPLIT_FLOATS'] = '1'
+        if 'autodict' not in build_modes:
+            os.environ['AFL_LLVM_LAF_TRANSFORM_COMPARES'] = '1'
+    # enable auto dictionary for LTO
+    if 'autodict' in build_modes:
+        os.environ['AFL_LLVM_LTO_AUTODICTIONARY'] = '1'
+
+    os.environ['FUZZER_LIB'] = '/libAFLDriver.a'
+
+    # Some benchmarks like lcms
+    # (see: https://github.com/mm2/Little-CMS/commit/ab1093539b4287c233aca6a3cf53b234faceb792#diff-f0e6d05e72548974e852e8e55dffc4ccR212)
+    # fail to compile if the compiler outputs things to stderr in unexpected
+    # cases. Prevent these failures by using AFL_QUIET to stop afl-clang-fast
+    # from writing AFL specific messages to stderr.
+    os.environ['AFL_QUIET'] = '1'
+
+    src = os.getenv('SRC')
+    work = os.getenv('WORK')
+    with utils.restore_directory(src), utils.restore_directory(work):
+        # Restore SRC to its initial state so we can build again without any
+        # trouble. For some OSS-Fuzz projects, build_benchmark cannot be run
+        # twice in the same directory without this.
+        utils.build_benchmark()
+
+    if 'cmplog' in build_modes and 'qemu' not in build_modes:
+
+        # CmpLog requires an build with different instrumentation.
+        new_env = os.environ.copy()
+        new_env['AFL_LLVM_CMPLOG'] = '1'
+
+        # For CmpLog build, set the OUT and FUZZ_TARGET environment
+        # variable to point to the new CmpLog build directory.
+        cmplog_build_directory = get_cmplog_build_directory(build_directory)
+        os.mkdir(cmplog_build_directory)
+        new_env['OUT'] = cmplog_build_directory
+        fuzz_target = os.getenv('FUZZ_TARGET')
+        if fuzz_target:
+            new_env['FUZZ_TARGET'] = os.path.join(cmplog_build_directory,
+                                                  os.path.basename(fuzz_target))
+
+        print('Re-building benchmark for CmpLog fuzzing target')
+        utils.build_benchmark(env=new_env)
+
+    shutil.copy('/afl/afl-fuzz', build_directory)
+
+
+def fuzz(input_corpus, output_corpus, target_binary, flags=tuple()):
+    """Run fuzzer."""
+    # Calculate CmpLog binary path from the instrumented target binary.
+    target_binary_directory = os.path.dirname(target_binary)
+    cmplog_target_binary_directory = (
+        get_cmplog_build_directory(target_binary_directory))
+    target_binary_name = os.path.basename(target_binary)
+    cmplog_target_binary = os.path.join(cmplog_target_binary_directory,
+                                        target_binary_name)
+
+    afl_fuzzer.prepare_fuzz_environment(input_corpus)
+    # decomment this to enable libdislocator
+    # os.environ['AFL_ALIGNED_ALLOC'] = '1' # align malloc to max_align_t
+    # os.environ['AFL_PRELOAD'] = '/afl/libdislocator.so'
+
+    flags = list(flags)
+    if os.path.exists(cmplog_target_binary):
+        flags += ['-c', cmplog_target_binary]
+    if 'ADDITIONAL_ARGS' in os.environ:
+        flags += os.environ['ADDITIONAL_ARGS'].split(' ')
+
+    # needed for LTO mode to run c++ targets
+    os.environ['LD_LIBRARY_PATH'] = '/out'
+
+    afl_fuzzer.run_afl_fuzz(input_corpus,
+                            output_corpus,
+                            target_binary,
+                            additional_flags=flags)
diff --git a/fuzzers/aflplusplus_shmem/runner.Dockerfile b/fuzzers/aflplusplus_shmem/runner.Dockerfile
@@ -0,0 +1,15 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM gcr.io/fuzzbench/base-runner