Use bundled libFuzzer runtime from clang instead of building it locally

[jonathanmetzman]

@kcc points out that building locally is causing performance issues since we aren't using the correct build flags.

I've experimented with linking fuzz targets against the bundled runtime instead of the locally built one.
I'm hoping to transparently do this so that $LIB_FUZZING_ENGINE is the bundled runtime instead of a locally built one without needing any changes in projects' build processes.

However, @Dor1s and I found that we need to use -lc++abi when linking directly against the bundled libFuzzer. Is this expected?
Is there a way to force everyone linking against libFuzzer to also link against c++abi (maybe the libcxxabi.a and libclang_rt.fuzzer-x86_64.a libraries can be combined)?
Adding -lc++abi to CXXFLAGS/CFLAGS sort of works, but causes a lot of compiler warnings when the flags are used just for compilation and not linking.

cc @morehouse

[morehouse]

Wouldn't -fsanitize=fuzzer automatically get the bundled runtime and correct flags?

[jonathanmetzman]

Most projects already use -lFuzzingEngine and I wanted to make the change transparent.
So I wan't planning on using -fsanitize=fuzzer

[jonathanmetzman]

I suppose we could make libFuzzingEngine.a a dummy library so that -lFuzzingEngine doesn't fail and use -fsanitize=fuzzer

[jonathanmetzman]

Also, wouldn't -fsanitize=fuzzerbreak projects that build binaries with main functions as part of their build process?

[kcc]

We need to use -fsanitize=fuzzer-no-link at compile time (as we do already) and -fsanitize=fuzzer at link time.

Having -lFuzzingEngine was probably a bad idea, we should have enforced the usage of $LIB_FUZZING_ENGINE instead.
We can still do it now: replace -lFuzzingEngine with $LIB_FUZZING_ENGINE where we can, and provide FuzzingEngine.a for compatibility with the build files we don't have.

Then, just set LIB_FUZZING_ENGINE to -fsanitize=fuzzer

[jonathanmetzman]

Then, just set LIB_FUZZING_ENGINE to -fsanitize=fuzzer

I guess a simple replace wont work for projects that currently pass -lFuzzingEngine directly to the linker instead of to the compiler?

[kcc]

no, if we also profile FuzzingEngine.a for compatibility (FuzzingEngine.a will have to be copied from the llvm build dir)

[jonathanmetzman]

But won't they need to be linked against libc++abi.a too?
In any case, I don't even know how many projects do this so we can see later how big of a problem this is.

[jonathanmetzman]

Giving this another shot.
For some reason it seems like even with -fsanitize=fuzzer compilation will fail unless we use -lc++abi.

Perhaps this is because of the options used to compile libc++?

See this for example (summary below):

I  Step #4: CFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link 
I  Step #4: CXXFLAGS=-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ 
I  Step #4: --------------------------------------------------------------- 
I  Step #4: + clang++ -c -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ skcms.cc -DIS_FUZZING_WITH_LIBFUZZER 
I  Step #4: + clang -c -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link fuzz/fuzz_iccprofile_info.c fuzz/fuzz_iccprofile_atf.c fuzz/fuzz_iccprofile_transform.c -DIS_FUZZING_WITH_LIBFUZZER 
I  Step #4: + clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link -stdlib=libc++ skcms.o fuzz_iccprofile_info.o -fsanitize=fuzzer -o /workspace/out/address/iccprofile_info 
I  Step #4: /usr/local/lib/clang/9.0.0/lib/linux/libclang_rt.asan_cxx-x86_64.a(ubsan_type_hash_itanium.cc.o): In function `isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long) [clone .constprop.0]': 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:126: undefined reference to `typeinfo for __cxxabiv1::__si_class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:126: undefined reference to `typeinfo for __cxxabiv1::__class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:130: undefined reference to `typeinfo for __cxxabiv1::__vmi_class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:130: undefined reference to `typeinfo for __cxxabiv1::__class_type_info' 
I  Step #4: /usr/local/lib/clang/9.0.0/lib/linux/libclang_rt.asan_cxx-x86_64.a(ubsan_type_hash_itanium.cc.o): In function `findBaseAtOffset(__cxxabiv1::__class_type_info const*, long)': 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:163: undefined reference to `typeinfo for __cxxabiv1::__si_class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:163: undefined reference to `typeinfo for __cxxabiv1::__class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:167: undefined reference to `typeinfo for __cxxabiv1::__vmi_class_type_info' 
I  Step #4: /usr/local/lib/clang/9.0.0/lib/linux/libclang_rt.asan_cxx-x86_64.a(ubsan_type_hash_itanium.cc.o): In function `findBaseAtOffset': 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:167: undefined reference to `typeinfo for __cxxabiv1::__vmi_class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:167: undefined reference to `typeinfo for __cxxabiv1::__vmi_class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:167: undefined reference to `typeinfo for __cxxabiv1::__vmi_class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:167: undefined reference to `typeinfo for __cxxabiv1::__vmi_class_type_info' 
I  Step #4: /usr/local/lib/clang/9.0.0/lib/linux/libclang_rt.asan_cxx-x86_64.a(ubsan_type_hash_itanium.cc.o):/src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:167: more undefined references to `typeinfo for __cxxabiv1::__vmi_class_type_info' follow 
I  Step #4: /usr/local/lib/clang/9.0.0/lib/linux/libclang_rt.asan_cxx-x86_64.a(ubsan_type_hash_itanium.cc.o): In function `__ubsan::checkDynamicType(void*, void*, unsigned long)': 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:233: undefined reference to `typeinfo for __cxxabiv1::__class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:233: undefined reference to `typeinfo for std::type_info' 
I  Step #4: /usr/local/lib/clang/9.0.0/lib/linux/libclang_rt.asan_cxx-x86_64.a(ubsan_type_hash_itanium.cc.o): In function `findBaseAtOffset': 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:159: undefined reference to `typeinfo for __cxxabiv1::__si_class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:159: undefined reference to `typeinfo for __cxxabiv1::__class_type_info' 
I  Step #4: /src/llvm/projects/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cc:167: undefined reference to `typeinfo for __cxxabiv1::__vmi_class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(private_typeinfo.cpp.o): In function `__cxxabiv1::__class_type_info::can_catch(__cxxabiv1::__shim_type_info const*, void*&) const': 
I  Step #4: private_typeinfo.cpp:(.text._ZNK10__cxxabiv117__class_type_info9can_catchEPKNS_16__shim_type_infoERPv+0x37): undefined reference to `typeinfo for __cxxabiv1::__class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(private_typeinfo.cpp.o): In function `__cxxabiv1::__pointer_type_info::can_catch(__cxxabiv1::__shim_type_info const*, void*&) const': 
I  Step #4: private_typeinfo.cpp:(.text._ZNK10__cxxabiv119__pointer_type_info9can_catchEPKNS_16__shim_type_infoERPv+0x1cb): undefined reference to `typeinfo for __cxxabiv1::__class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_personality.cpp.o): In function `__cxa_call_unexpected': 
I  Step #4: cxa_personality.cpp:(.text.__cxa_call_unexpected+0x164): undefined reference to `vtable for std::bad_exception' 
I  Step #4: cxa_personality.cpp:(.text.__cxa_call_unexpected+0x16e): undefined reference to `typeinfo for std::bad_exception' 
I  Step #4: /usr/local/bin/../lib/libc++.a(stdlib_exception.cpp.o): In function `std::bad_exception::~bad_exception()': 
I  Step #4: stdlib_exception.cpp:(.text._ZNSt13bad_exceptionD2Ev+0x3): undefined reference to `vtable for std::bad_exception' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_default_handlers.cpp.o): In function `demangling_terminate_handler()': 
I  Step #4: cxa_default_handlers.cpp:(.text._ZL28demangling_terminate_handlerv+0xa7): undefined reference to `typeinfo for std::exception' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_demangle.cpp.o):(.data.rel.ro._ZTIN12_GLOBAL__N_116itanium_demangle4NodeE+0x0): undefined reference to `vtable for __cxxabiv1::__class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_demangle.cpp.o):(.data.rel.ro._ZTIN12_GLOBAL__N_116itanium_demangle13NodeArrayNodeE+0x0): undefined reference to `vtable for __cxxabiv1::__si_class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_demangle.cpp.o):(.data.rel.ro._ZTIN12_GLOBAL__N_116itanium_demangle9DotSuffixE+0x0): undefined reference to `vtable for __cxxabiv1::__si_class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_demangle.cpp.o):(.data.rel.ro._ZTIN12_GLOBAL__N_116itanium_demangle17VendorExtQualTypeE+0x0): undefined reference to `vtable for __cxxabiv1::__si_class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_demangle.cpp.o):(.data.rel.ro._ZTIN12_GLOBAL__N_116itanium_demangle8QualTypeE+0x0): undefined reference to `vtable for __cxxabiv1::__si_class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_demangle.cpp.o):(.data.rel.ro._ZTIN12_GLOBAL__N_116itanium_demangle22ConversionOperatorTypeE+0x0): undefined reference to `vtable for __cxxabiv1::__si_class_type_info' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_demangle.cpp.o):(.data.rel.ro._ZTIN12_GLOBAL__N_116itanium_demangle20PostfixQualifiedTypeE+0x0): more undefined references to `vtable for __cxxabiv1::__si_class_type_info' follow 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_aux_runtime.cpp.o): In function `__cxa_bad_cast': 
I  Step #4: cxa_aux_runtime.cpp:(.text.__cxa_bad_cast+0x20): undefined reference to `typeinfo for std::bad_cast' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_aux_runtime.cpp.o): In function `__cxa_bad_typeid': 
I  Step #4: cxa_aux_runtime.cpp:(.text.__cxa_bad_typeid+0x20): undefined reference to `typeinfo for std::bad_typeid' 
I  Step #4: /usr/local/bin/../lib/libc++.a(cxa_aux_runtime.cpp.o): In function `__cxa_throw_bad_array_new_length': 
I  Step #4: cxa_aux_runtime.cpp:(.text.__cxa_throw_bad_array_new_length+0x20): undefined reference to `typeinfo for std::bad_array_new_length' 
I  Step #4: clang-9: error: linker command failed with exit code 1 (use -v to see invocation) 

[kcc]

@morehouse

[morehouse]

Not sure exactly what circumstances require an explicit -lc++abi, but I do see it mentioned in the libc++ docs: https://libcxx.llvm.org/docs/UsingLibcxx.html#using-libc-on-linux.

Possibly dumb question: Any reason we can't add -lc++abi to LDFLAGS rather than CFLAGS?

[morehouse]

... or add it to LIB_FUZZING_ENGINE?