Full libfuzzer shimming (for cargo-fuzz libfuzzer alternative and other use cases)

[addisoncrump]

Seeing as how libfuzzer has entered maintenance mode, we should provide a full libfuzzer alternative which stays up to date with modern fuzzing standards.

To this end, this PR seeks to offer libafl_libfuzzer, a full drop-in replacement for libfuzzer with support for the most common flags and sancov settings. Specifically, all of the things supported by cargo-fuzz. We can provide this shim to cargo-fuzz via environmental variable in libfuzzer-sys or, in the future, an init flag in cargo-fuzz itself.

We need to implement corpus merging (fairly straightforward with IndexesLenTimeMinimizingScheduler or cmin), crash minification (doable with tmin + InProcessForkExecutor), and a basic fuzzing runtime (optionally with dict support). cargo-fuzz uses many of the available sancov features, so we need some additional support for __san*cov items. As of writing, we have the following undefined references:

$ cargo fuzz build -s none |& grep -Eo "undefined reference to.*'" | sort -u 
undefined reference to `LLVMFuzzerMutate'
undefined reference to `__sancov_lowest_stack'
undefined reference to `__sanitizer_cov_pcs_init'
undefined reference to `__sanitizer_cov_trace_pc_indir'

I'm unfamiliar with __san*cov features, so I could use some help in developing that support.

I think this would be a good addition to the 0.9 release as well.

[domenukk]

LibAFL/fuzzers/libafl_atheris/src/lib.rs

Line 107 in d2985c5

pub fn LLVMFuzzerRunDriver(

Can we add support for the run FN and replace most of atheris, too?

[addisoncrump]

LibAFL/fuzzers/libafl_atheris/src/lib.rs

Line 107 in d2985c5

pub fn LLVMFuzzerRunDriver(

Can we add support for the run FN and replace most of atheris, too?

I'm unfamiliar with this aspect of the libfuzzer functionality. I'm sure we can replace it, though.

[domenukk]

LibAFL/fuzzers/libafl_atheris/src/lib.rs

Line 107 in d2985c5

pub fn LLVMFuzzerRunDriver(

Can we add support for the run FN and replace most of atheris, too?

I'm unfamiliar with this aspect of the libfuzzer functionality. I'm sure we can replace it, though.

It's a way for the target to start fuzzing at another point in time, so an own main, instead of using libfuzzer's main. We can probably move all code to this method and call this fn from a weak main

[domenukk]

AFLplusplus/AFLplusplus#1515

See this issue for the run method, too

[domenukk]

Reading through the documentation at https://clang.llvm.org/docs/SanitizerCoverage.html#tracing-pcs-with-guards it looks like this is pretty straight forward, we just want a portion of the coverage map for indirect branches and then map them as feedback.
Also, it looks like the function is called __sanitizer_cov_trace_pc_indirect(void *callee) in newer compilers (or in some versions and the docs are old?) so maybe we want to export both

[domenukk]

Support can be as easy as exposing this value to rust via an fn - then we can build an observer that observes stack depth of a run, quasi for free

[Mrmaxmeier]

Hi,
I wanted to try this but I'm seeing a bunch of "multiple definition" linker errors for Rust runtime symbols (__rdl_alloc, rust_eh_personality, ...).
I'm setting CUSTOM_LIBFUZZER_PATH to <LibAFL>/target/debug/build/libafl_libfuzzer-00bdb9ec1c4f4eed/out/runtime-target/x86_64-unknown-linux-gnu/release/libafl_libfuzzer_runtime.a (without the rust-fuzz/libfuzzer#103 fix applied).

Am I holding it wrong? (Compiling to a cdylib instead of staticlib or using the .rlib instead of the .a doesn't seem to work either)

[addisoncrump]

Hey, thanks for giving this a go. Try building it from within libafl_libfuzzer_runtime directly instead? It sounds like what your doing should work, so if you're still having issues I can investigate with you in Discord. Also, make sure you're building with the same toolchain as your default (e.g., avoid +nightly or +stable flags, I haven't sorted out a bug there just yet).

[Mrmaxmeier]

Building libafl_libfuzzer_runtime directly works 👍
The -Zbuild-std line in the wrapper build script seems to break stuff on my machine 🤷

[addisoncrump]

07abe31 is blocked by #1067

[addisoncrump]

I don't understand how to use the whole-archive feature (the original code results in duplicate symbols for __rust_alloc and friends with cargo-fuzz), but my commit fixes the check step in CI. 🙂

This is probably just an error on my part... mimalloc doesn't play well with being linked separately.

[tokatoka]

i guess if you launch the manager everytime, then the timestamp is not preserved across restarts

[domenukk]

Maybe fix this after merging/open an issue?

[domenukk]

error[E0133]: call to unsafe function is unsafe and requires unsafe function or block
   --> libafl/src/schedulers/probabilistic_sampling.rs:192:13
    |
192 |             libafl_bolts::serdeany::RegistryBuilder::register::<super::ProbabilityMetadata>();
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ call to unsafe function
    |
    = note: consult the function's documentation for information on how to avoid undefined behavior

error: aborting due to previous error

Seems like we're close, though

[jasikpark]

🥳

[ndrewh]

      Finished release [optimized + debuginfo] target(s) in 4m 21s
  LLVM ERROR: Invalid encoding
  thread 'main' panicked at 'Couldn't create runtime archive!', libafl_libfuzzer/build.rs:86:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo check --manifest-path libafl_libfuzzer/Cargo.toml --no-default-features --features whole-archive` (exit status: 101)
Error: Process completed with exit code 1.

That's a new one..

@addisoncrump any ideas how to fix this?

I ran into this. It turns out that nm behavior differs depending on your system LLVM, and that uninstalling your system LLVM fixes the issue.

backtrace to the error (TIL that nm just invokes your system LLVM)

#5  0x00007fffe898d723 in llvm::report_fatal_error(llvm::Twine const&, bool) () from /lib/x86_64-linux-gnu/libLLVM-14.so.1
#6  0x00007fffe898d556 in llvm::report_fatal_error(char const*, bool) () from /lib/x86_64-linux-gnu/libLLVM-14.so.1
#7  0x00007fffe94a79ca in llvm::BitstreamCursor::ReadAbbrevRecord() () from /lib/x86_64-linux-gnu/libLLVM-14.so.1
#8  0x00007fffe942dfce in ?? () from /lib/x86_64-linux-gnu/libLLVM-14.so.1
#9  0x00007fffe94341bd in llvm::getBitcodeFileContents(llvm::MemoryBufferRef) () from /lib/x86_64-linux-gnu/libLLVM-14.so.1
#10 0x00007fffea0efc75 in llvm::object::readIRSymtab(llvm::MemoryBufferRef) () from /lib/x86_64-linux-gnu/libLLVM-14.so.1
#11 0x00007fffe9f663d2 in llvm::lto::InputFile::create(llvm::MemoryBufferRef) () from /lib/x86_64-linux-gnu/libLLVM-14.so.1
#12 0x00007fffee4ed22a in ?? () from /usr/bin/../bin/../lib/bfd-plugins/LLVMgold-14.so
#13 0x00007ffff7f497c2 in ?? () from /lib/x86_64-linux-gnu/libbfd-2.38-system.so
#14 0x00007ffff7f49a1c in ?? () from /lib/x86_64-linux-gnu/libbfd-2.38-system.so
#15 0x00007ffff7e8790c in bfd_check_format_matches () from /lib/x86_64-linux-gnu/libbfd-2.38-system.so

I made the grave error of having LLVM 14 installed on my machine. Uninstalling LLVM 14 fixed the issue -- libbfd instead chose the LLVMgold-15.so plugin (nope -- i have no idea where it makes this decision) which... uhhh... doesn't seem to invoke my system LLVM at all and prints the symbols successfully.

[tokatoka]

Some symbols we are using in libafl_libfuzzer are missing in LLVM 14, so it doesn't work on llvm <= 14

[ndrewh]

@tokatoka I think you misunderstand. build.rs invokes nm, which was hitting an error depending on the system LLVM:

  LLVM ERROR: Invalid encoding

Ideally, the behavior of this crate would not depend on whether or not /lib/x86_64-linux-gnu/libLLVM-14.so.1 exists on the system. (note: /lib/x86_64-linux-gnu/libLLVM-15.so.1 also exists on the system). IMO this is probably an upstream issue, but we probably shouldn't be using the system nm here to begin with.

[ndrewh]

In theory i think the llvm-nm that ships with cargo's llvm-tools component should work, but it hits a different encoding error regardless of system LLVM.

[tokatoka]

my point is even if you avoid that issue, the fuzzer wouldn't have linked using llvm 14.