Skip to content
Structure-aware, in-process, coverage-guided, evolutionary fuzzing engine for Rust functions.
Rust C Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/workflows
cargo-fuzzcheck
fuzzcheck
fuzzcheck_arg_parser
fuzzcheck_input Run cargo fmt Aug 27, 2019
usage_tests
.gitignore
Cargo.lock
Cargo.toml
LICENSE.txt
readme.md
rustfmt.toml
status.txt

readme.md

Fuzzcheck

I made Fuzzcheck in my free time during my summer vacation. It is an improved port of FuzzCheck for Swift, which I wrote a year ago and discussed in a talk at the Functional Swift conference. There are many, many ways in which it could be improved such that it becomes a powerful, easy-to-use tool for any Rust programmer. I would love to keep working on it, but I am now back to university, and I find it hard to justify spending a significant amount of time on it. If you would like me to keep developing Fuzzcheck or help your company use it, please hire me and help me pay for my studies :)

Fuzzcheck is a structure-aware, in-process, coverage-guided, evolutionary fuzzing engine for Rust functions.

Its main aim is to be used as the input generator of property-based tests. Detecting security flaws in an application is a non-goal.

Given a function test: (T) -> Bool, it tries to find a value of type T that fails the test or leads to a crash.

Unlike other fuzzing engines, it doesn't work with bitstrings but instead works with values of any type T directly. The complexity of the inputs and the way to mutate them is given by functions defined by the user.

Fuzzcheck works by maintaining a pool of test inputs and ranking them using the complexity of the input and the uniqueness of the code coverage caused by test(input). From that pool, it selects a high-ranking input, mutates it, and runs the test function again. If the new mutated input has an interesting code coverage then it is added to the pool, otherwise, Fuzzcheck tries again with a different input and mutation.

In pseudocode:

loop {
    let mut input = pool.select();
    mutate(&mut input);

    let analysis = analyze(test, &input);

    match analysis {
        Failed => reportFailure(input),
        Interesting(score) => pool.add(input, score),
        NotInteresting => continue
    }
}

Example

This repo contains a very simple test involving a graph data structure. You can quickly run it on your computer:

fn test(graph: &Graph<i8>) -> bool {
    if 
    graph.nodes.len() == 8 && 
    graph.nodes[0].data == 63 && 
    graph.nodes[1].data == 3 &&
    graph.nodes[2].data == -56 &&
    graph.nodes[3].data == 100 &&
    graph.nodes[4].data == -100 &&
    graph.nodes[5].data == -78 &&
    graph.nodes[6].data == 46 &&
    graph.nodes[7].data == 120 &&
    
    graph.nodes[0].edges.len() == 2 && 
    graph.nodes[0].edges[0] == 1 && 
    graph.nodes[0].edges[1] == 2 && 
    graph.nodes[1].edges.len() == 2 && 
    graph.nodes[1].edges[0] == 3 && 
    graph.nodes[1].edges[1] == 4 && 
    graph.nodes[2].edges.len() == 2 && 
    graph.nodes[2].edges[0] == 5 && 
    graph.nodes[2].edges[1] == 6 && 
    graph.nodes[3].edges.len() == 1 && 
    graph.nodes[3].edges[0] == 7 && 
    graph.nodes[4].edges.len() == 0 && 
    graph.nodes[5].edges.len() == 0 && 
    graph.nodes[6].edges.len() == 0 && 
    graph.nodes[7].edges.len() == 0 {
        return false
    }
    true
}

fn main() {
    let i8_gen = IntegerGenerator::<i8>::default();
    let graph_gen = GraphGenerator::new(i8_gen);

    // will very quickly find the graph that satisfies all the conditions in th test function
    let _ = fuzzer::launch(test, graph_gen);
}

Usage

The first step is to install the cargo-fuzzcheck executable using cargo nightly.

cargo +nightly install --git https://github.com/loiclec/fuzzcheck-rs

Then, somewhere else, create a new cargo library crate. It will contain the library code that you want to fuzz-test. Also do not forget to set the rust version to nightly.

cargo new --lib my_library
cd my_library
rustup override set nightly

Then, run cargo fuzzcheck init to install the fuzzcheck library and initialize a fuzz folder that will contain all future fuzz tests.

cargo fuzzcheck init

An executable script was created at fuzz/fuzz_targets/target1.rs. It contains a basic fuzz test that works with values of type Vec<u8>.

extern crate my_library;

extern crate fuzzcheck;
use fuzzcheck::fuzzer;

extern crate fuzzcheck_input;
use fuzzcheck_input::integer::IntegerGenerator;
use fuzzcheck_input::vector::VectorGenerator;

fn test(input: &Vec<u8>) -> bool {
    // property test goes here
    if 
        input.len() > 7 &&
        input[0] == 0 &&
        input[1] == 167 &&
        input[2] == 200 &&
        input[3] == 103 &&
        input[4] == 56 &&
        input[5] == 78 &&
        input[6] == 2 &&
        input[7] == 254
    {
        false
    }
    else {
        true
    }
}

fn main() {
    let u8_gen = IntegerGenerator::<u8>::new();
    let vec_gen = VectorGenerator::new(u8_gen);
    
    let _ = fuzzer::launch(test, vec_gen);
}

Note that while the input is of type Vec<u8>, it could equally easily be anything such as String, HashMap<T, U>, etc.

You can already try launching this test:

cargo fuzzcheck run target1 fuzz

This starts a loop that will stop when a failing test has been found.

A line will be printed whenever a newsworthy event happened, along with some statistics. For example:

NEW     180086  score: 493      pool: 48        exec/s: 132713  cplx: 79792
  • NEW means that a new input was added to the pool of interesting inputs
  • 180086 is the number of iterations that were performed so far
  • score: 493 is a measure of the total code coverage caused by all inputs in the pool
  • pool: 48 is the number of inputs in the pool
  • exec/s: 132713 is the average number of iterations performed every second
  • cplx: 79792 is the average complexity of the inputs in the pool

When a failing test has been found, the following is printed:

================ TEST FAILED ================
188241  score: 495      pool: 51        exec/s: 132659  cplx: 81373
Saving at "./fuzz/fuzz_targets/target1/artifacts/1c10daa13e9b1721.json"

Here, the path to the artifact file is ./fuzz/fuzz_targets/target1/artifacts/1c10daa13e9b1721.json. It contains a JSON-encoding of the input that failed the test.

[0, 167, 200, 103, 56, 78, 2, 254]

Moreover, the fuzzer can maintain a copy of its input pool in the file system, which is located by default at fuzz_targets/<target>/fuzz-corpus/. Fuzzing corpora are useful to kick-start a fuzzing process by providing a list of known interesting inputs. If you try to run the fuzzer again, you will see that it finds the problematic input much quicker. This is because it first read the values written inside fuzz-corpus and used them as starting points.

Minifying failing test inputs

Fuzzcheck can also be used to minify a large input that fails a test.

Let's say you have a file crash.json containing an input that you would like to minify:

[0,78,56,2,76,7,100,102,102,0,0,78,56,2,76,
7,100,102,102,0,234,169,95,18,254,102,81,
41,212,142,0,78,56,2,76,7,100,102,102,0]

Launch cargo-fuzzcheck run on your target with the tmin command and an --input-file flag.

cargo fuzzcheck run target1 tmin --input-file "artifacts/crash.json"

This will repeatedly launch the fuzzer in “minify” mode and save the artifacts in the folder artifacts/crash.minified. The name of each artifact will be prefixed with the complexity of its input. For example, crash.minified/800--fe958d4f003bd4f5.json has a complexity of 8.00.

You can stop the minifying fuzzer at any point and look for the least complex input in the crash.minified folder.

Creating an InputGenerator

If you would like to fuzz-test your own custom type, you will have to create an input generator for it. You can do so by creating a type that conforms to the InputGenerator trait.

pub trait InputGenerator {
    type Input: Clone;

    fn hash<H>(input: &Self::Input, state: &mut H) where H: Hasher;

    fn base_input() -> Self::Input;
    fn complexity(input: &Self::Input) -> f64;
    
    fn new_input(&mut self, max_cplx: f64) -> Self::Input;

    fn mutate(&mut self, input: &mut Self::Input, spare_cplx: f64) -> bool;

    fn from_data(data: &[u8]) -> Option<Self::Input>;
    fn to_data(input: &Self::Input) -> Vec<u8>;
}
  • Input is the type being tested, it must be cloneable.

  • hash should be implemented in the same way as the hash associated function of the Hash trait.

  • base_input is the simplest value of type Input that you can think of. It may be 0 for numbers or an empty vector for Vec.

  • complexity returns a float that estimates how “complex” the input is. For an integer, this might be the number of bytes used to represent it. For an array, it might be the sum of complexities of each of its elements.

  • new_input returns a random input with a complexity smaller than max_cplx

  • mutate mutates the given input without increasing its complexity by more than spare_cplx (otherwise, the fuzzer will ignore the result and skip the current iteration, which is not too bad but slows it down). The mutation should ideally be small, but meaningful. For example, it could:

    • append a random element to an array
    • mutate a random element in an array
    • subtract a small constant from an integer
    • change an integer to 0, or its minimum/maximum value
    • replace a substring by a keyword relevant to the test function
    • add a node to a graph data structure, and connect it to a random node
  • from_data and to_data decode/encode the input. They do not have to be fast or memory efficient. For example, a simple implementation of to_data could be:

    fn to_data(input: &Self::Input) -> Vec<u8> {
        serde_json::to_vec(input).unwrap()
    }

Previous work on fuzzing engines

As far as I know, evolutionary, coverage-guided fuzzing engines were popularized by American Fuzzy Lop (AFL).
Fuzzcheck is also evolutionary and coverage-guided.

Later on, LLVM released its own fuzzing engine, libFuzzer, which is based on the same ideas as AFL, but it uses Clang’s SanitizerCoverage and is in-process (it lives in the same process as the program being fuzz-tested.
Fuzzcheck is also in-process and also uses SanitizerCoverage.

Both AFL and libFuzzer work by manipulating bitstrings (e.g. 1011101011). However, many programs work on structured data, and mutations at the bitstring level may not map to meaningful mutations at the level of the structured data. This problem can be partially addresses by using a compact binary encoding such as protobuf and providing custom mutation functions to libFuzzer that work on the structured data itself. This is a way to perform “structure-aware fuzzing” (talk, tutorial).

Fuzzcheck is also structure-aware, but unlike previous attempts at structure-aware fuzzing, it doesn't use an intermediary binary encoding such as protobuf. Instead, it directly works with the typed values in-process. This is better in at least three ways. First, it is faster because there is no need to encode and decode inputs at each iteration. Second, the complexity of the input is given by a user-defined function, which will be more accurate than counting the bytes of the protobuf encoding. Third, the artifact files and the fuzzing corpora can be JSON-encoded, which is more user-friendly than protobuf.

As I was developing Fuzzcheck, a few researchers developed Fuzzchick for Coq (paper). It is a coverage-guided property-based testing tool implemented as an extension to Quickchick.

You can’t perform that action at this time.