Skip to content
Permalink
Browse files

Introduce ClusterFuzz support (#15)

With this commit I've now enabled clusterfuzz support for the project, running at https://bughunt.appspot.com/. The use of AFL is now removed in favor of cargo-fuzz/libFuzzer for easy compatibility with clusterfuzz. This has made the README instructions more straightforward, which hopefully will help new contributors to the project.

The clusterfuzz setup consumes actual funds, though for now we're operating out of credits. This project will need to sort out funding at some point, if the clusterfuzz setup proves worth keeping.
  • Loading branch information...
blt committed Mar 10, 2019
1 parent f5c86a4 commit e7240771fe8d179200994092b314e05e2939a9f4
@@ -4,4 +4,5 @@ Cargo.lock
resources/
hfuzz_*
out/
in/
in/
ci/auth.json
@@ -1,33 +1,11 @@
language: rust
cache: cargo

env:
global:
- QUICKCHECK_GENERATOR_SIZE=10000
- QUICKCHECK_TESTS=10000

matrix:
include:
- rust: stable
before_script:
- rustup component add rustfmt
- rustup component add clippy
script:
- cargo fmt -- --check
- cargo clippy
- cargo test
- rust: stable
script:
- cargo test --release
- rust: beta
script:
- cargo test
- rust: beta
script:
- cargo test --release
- rust: nightly
script:
- cargo test
- rust: nightly
script:
- cargo test --release
rust:
- nightly
script:
- cargo install --force cargo-fuzz
- "./ci/builds.sh"
- "./ci/upload_builds.sh"
before_install:
- openssl aes-256-cbc -K $encrypted_edc4a4aaa9a0_key -iv $encrypted_edc4a4aaa9a0_iv
-in ci/auth.json-enc -out ci/auth.json -d
@@ -4,23 +4,12 @@ version = "0.1.0-pre"
authors = ["Brian L. Troutwine <brian@troutwine.us>"]

[dependencies]
afl = "0.4"
arbitrary = "0.2"
bh_alloc = "0.2.4"
strum = "0.13.0"
strum_macros = "0.13.0"

[[bin]]
path = "src/bin/stdlib/str/repeat.rs"
name = "str_repeat"

[[bin]]
path = "src/bin/stdlib/collections/hash_map.rs"
name = "hash_map"

[[bin]]
path = "src/bin/stdlib/collections/vec_deque.rs"
name = "vec_deque"
[dependencies.libfuzzer-sys]
git = "https://github.com/rust-fuzz/libfuzzer-sys.git"

[patch.crates-io]
strum_macros = { git = "https://github.com/Peternator7/strum" }
@@ -21,48 +21,23 @@ for crashes.
## Running the Suite

Running the tests takes a little leg work. The project performs model-based
fuzzing, which means the tests are driven by a fuzzer, AFL in particular. We've
written about the general approach
[here](https://blog.troutwine.us/2018/10/08/hunting-for-bugs-in-rust/).
fuzzing, which means the tests are driven by a fuzzer, cargo-fuzz (libFuzzer) in
particular. We've written about the general approach
[here](https://blog.troutwine.us/2018/10/08/hunting-for-bugs-in-rust/). Since
this post we've switch from AFL to libFuzzer but the broad details remain the
same.

The available targets are listed out in [`Cargo.toml`], the binaries of the
project. Say you want to run the `str::repeat` target. Make sure you've got AFL
installed by running `cargo install afl`. That done, create input and output
directories for the fuzzer. The input directory influences what the fuzzer
initially uses to populate it's testcase pool. The output directory will hold
crashes, timeout etc data. Inputs have a huge influence on the running behaviour
of a target but it's not straightforward to know what input you should
supply. This is, uh, an open area of research.

We'll create an input directory filled with not-so-great data

```
> mkdir -p /tmp/repeat/in
> date >> /tmp/repeat/in/0000
> date >> /tmp/repeat/in/0001
> date >> /tmp/repeat/in/0002
> date >> /tmp/repeat/in/0003
```

and an output directory:
The available targets are listed out in [`fuzz/Cargo.toml`], the binaries of the
project. Say you want to run the `str::repeat` target. Make sure you've got
cargo-fuzz installed by running `cargo install cargo-fuzz`.

```
> mkdir -p /tmp/repeat/out
> cargo fuzz run str_repeat
```

You can place these anywhere on disk you'd like. This is just an example. Okay,
from the root of the project:

```
> cargo afl build
> cargo afl fuzz -i /tmp/repeat/in -o /tmp/repeat/out/ target/debug/str_repeat
```

A reasonable test run will take hours. With the flags used above the run will
proceed indefinitely. Please note that AFL is single-threaded and to exploit
multi-core systems you'll need to spawn additional worker processes. This is
documented in the [AFL
project](https://github.com/mirrorer/afl/blob/master/docs/parallel_fuzzing.txt).
A reasonable test run will take hours and as configured the above run will
execute forever. Give the flag `--help` to `cargo fuzz` to see its options
relating to runtime constriction, corpus definition etc.

### Why does this run outside of Rust itself?

@@ -71,20 +46,25 @@ project is something anyone would go for and, working here as an external
project, we can avoid needing to fiddle with toolchains and longish build
cycles. Downside is, the std data structures we're testing don't have any
sanitizers turned on etc on account of the project is run against the usual Rust
releases.
release.

## Contributing

Writing QuickCheck models can be slow work and contributions are _very_ welcome,
either introducing new models into the project or extending existing ones. Once
the project is a little more advanced donations of computing resources will
also be welcome. Writing QuickCheck models can be slow but, boy, running them is
no joke.
either introducing new models into the project or extending existing ones. We
have an experimental [clusterfuzz](https://github.com/google/clusterfuzz) setup
running and if you have credits to donate that would be most welcome. I intend
to document project balances, money needs once they are clear.

### Would you take CI help?

Yes! It'd be really nifty if this project could be run automatically against
every nightly, for instance, and flag when issues are discovered.
Yes! Right now we have a folder `ci/` which has the build scripts used in
`.travis.yml`. We're producing test binaries and feeding them directly into the
clusterfuzz setup the project has. Speaking of, I'll be adding configuration for
that cluster to this repository in the coming days.

Any improvements in the build pipeline, clusterfuzz configuration are most
welcome.

### Would you take documentation help?

@@ -100,5 +80,8 @@ available (itself a problem, kind of). In no certain order:
* ["How Rust’s standard library was vulnerable for years and nobody noticed"](https://medium.com/@shnatsel/how-rusts-standard-library-was-vulnerable-for-years-and-nobody-noticed-aebf0503c3d6)
* ["PropEr Testing"](https://propertesting.com/)
* ["Moonconf Papers"](https://blog.troutwine.us/2016/05/26/moonconf-papers/)
* ["Hybrid Fuzz Testing:Discovering Software Bugs viaFuzzing and Symbolic Execution"](http://reports-archive.adm.cs.cmu.edu/anon/2012/CMU-CS-12-116.pdf)
* ["QuickFuzz: An Automatic Random Fuzzer for Common File Formats"](https://people.seas.harvard.edu/~pbuiras/publications/QFHaskell2016.pdf)
* ["Angora: Efficient Fuzzing by Principled Search"](http://web.cs.ucdavis.edu/~hchen/paper/chen2018angora.pdf)

I, blt, am also happy to answer questions over email. I'm brian@troutwine.us.
BIN +2.27 KB ci/auth.json-enc
Binary file not shown.
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
set -o errexit
set -o pipefail
set -o nounset

source ci/common.sh

for TEST in ${TESTS}
do
# 'cargo fuzz' doesn't have a build and I haven't investigated how to build
# without the guiding hand of the tool. Instead, fuzz each test for one
# second which, incidentally, builds the target.
#
# This is a silly hack.
cargo fuzz run ${TEST} -- -max_total_time=1
done
@@ -0,0 +1 @@
TESTS="str_repeat vec_deque hash_map"
@@ -0,0 +1,18 @@
#!/usr/bin/env bash
set -o errexit
set -o pipefail
set -o nounset

source ci/common.sh

BUILD_DIR=fuzz/target/x86_64-unknown-linux-gnu/debug/

gcloud auth activate-service-account --key-file ci/auth.json
for TEST in ${TESTS}
do
TRGT=${BUILD_DIR}${TEST}.tar.gz
SRCMAP=${BUILD_DIR}${TEST}-${TRAVIS_BUILD_NUMBER}.srcmap.json
echo "{\"rustc\": {\"type\": \"git\", \"url\": \"https://github.com/rust-lang/rust.git\", \"rev\": \"`rustc --version | awk '{print substr(\$3,2)}'`\"}}" | python -m json.tool > ${SRCMAP}
tar zcf ${TRGT} ${BUILD_DIR}${TEST} ${SRCMAP}
gsutil cp ${TRGT} gs://builds.bughunt.appspot.com/${TEST}/${TEST}-${TRAVIS_BUILD_NUMBER}.tar.gz
done
@@ -0,0 +1,4 @@

target
corpus
artifacts
@@ -0,0 +1,31 @@
[package]
name = "bughunt-rust-fuzz"
version = "0.0.1"
authors = ["Automatically generated"]
publish = false

[package.metadata]
cargo-fuzz = true

[dependencies]
arbitrary = "0.2"
[dependencies.bughunt-rust]
path = ".."
[dependencies.libfuzzer-sys]
git = "https://github.com/rust-fuzz/libfuzzer-sys.git"

# Prevent this from interfering with workspaces
[workspace]
members = ["."]

[[bin]]
path = "fuzz_targets/stdlib/str/repeat.rs"
name = "str_repeat"

[[bin]]
path = "fuzz_targets/stdlib/collections/hash_map.rs"
name = "hash_map"

[[bin]]
path = "fuzz_targets/stdlib/collections/vec_deque.rs"
name = "vec_deque"
@@ -0,0 +1,108 @@
#![no_main]
#[macro_use]
extern crate libfuzzer_sys;
extern crate arbitrary;
extern crate bughunt_rust;

use arbitrary::*;
use bughunt_rust::stdlib::collections::hash_map::*;
use std::collections::HashMap;

fuzz_target!(|data: &[u8]| {
if let Ok(mut ring) = FiniteBuffer::new(data, 16_384) {
let hash_seed: u8 = if let Ok(hs) = Arbitrary::arbitrary(&mut ring) {
hs
} else {
return;
};
// Why is capacity not usize? We're very likely to request a
// capacity so large that the HashMap cannot allocate enough
// slots to store them, resulting in a panic when we call
// `with_capacity_and_hasher`. This is a crash, but an
// uninteresting one.
//
// We also request a low-ish capacity, all but guaranteeing we'll
// force reallocation during execution.
//
// See note on [`hash_map::Op::Reserve`] for details
let capacity: u8 = if let Ok(cap) = Arbitrary::arbitrary(&mut ring) {
cap
} else {
return;
};

let mut model: PropHashMap<u16, u16> = PropHashMap::new();
let mut sut: HashMap<u16, u16, BuildTrulyAwfulHasher> = HashMap::with_capacity_and_hasher(
capacity as usize,
BuildTrulyAwfulHasher::new(hash_seed),
);

while let Ok(op) = Arbitrary::arbitrary(&mut ring) {
match op {
Op::Clear => {
// Clearing a HashMap removes all elements but keeps
// the memory around for reuse. That is, the length
// should drop to zero but the capacity will remain the
// same.
let prev_cap = sut.capacity();
sut.clear();
model.clear();
assert_eq!(0, sut.len());
assert_eq!(sut.len(), model.len());
assert_eq!(prev_cap, sut.capacity());
}
Op::ShrinkToFit => {
// NOTE There is no model behaviour here
//
// After a shrink the capacity may or may not shift from
// the passed arg `capacity`. But, the capacity of the
// HashMap should never grow after a shrink.
//
// Similarly, the length of the HashMap prior to a
// shrink should match the length after a shrink.
let prev_len = sut.len();
let prev_cap = sut.capacity();
sut.shrink_to_fit();
assert_eq!(prev_len, sut.len());
assert!(sut.capacity() <= prev_cap);
}
Op::Get { k } => {
let model_res = model.get(&k);
let sut_res = sut.get(&k);
assert_eq!(model_res, sut_res);
}
Op::Insert { k, v } => {
let model_res = model.insert(k, v);
let sut_res = sut.insert(k, v);
assert_eq!(model_res, sut_res);
}
Op::Remove { k } => {
let model_res = model.remove(&k);
let sut_res = sut.remove(&k);
assert_eq!(model_res, sut_res);
}
Op::Reserve { n } => {
// NOTE There is no model behaviour here
if sut.capacity().checked_add(n as usize).is_some() {
sut.reserve(n as usize);
} // else { assert!(sut.try_reserve(*n).is_err()); }
}
}
// Check invariants
//
// `HashMap<K, V>` defines the return of `capacity` as
// being "the number of elements the map can hold
// without reallocating", noting that the number is a
// "lower bound". This implies that:
//
// * the HashMap capacity must always be at least the
// length of the model
assert!(sut.capacity() >= model.len());
// If the SUT is empty then the model must be.
assert_eq!(model.is_empty(), sut.is_empty());
// The length of the SUT must always be exactly the length of
// the model.
assert_eq!(model.len(), sut.len());
}
}
});
@@ -1,18 +1,14 @@
extern crate bh_alloc;
#![no_main]
#[macro_use]
extern crate afl;
extern crate libfuzzer_sys;
extern crate arbitrary;
extern crate bughunt_rust;

#[global_allocator]
static ALLOC: bh_alloc::BumpAlloc = bh_alloc::BumpAlloc::INIT;

use arbitrary::*;
use bughunt_rust::stdlib::collections::vec_deque::*;
use std::collections::VecDeque;

fn main() {
fuzz!(|data: &[u8]| {
fuzz_target!(|data: &[u8]| {
if let Ok(mut ring) = FiniteBuffer::new(data, 65_563) {
let capacity: u8 = if let Ok(cap) = Arbitrary::arbitrary(&mut ring) {
cap
@@ -115,5 +111,4 @@ fn main() {
assert_eq!(sut.back(), model.back());
}
}
})
}
});
Oops, something went wrong.

0 comments on commit e724077

Please sign in to comment.
You can’t perform that action at this time.