Distributed fuzzing using AFL.
AFL is a "fuzzer". You give it a target program, and it runs that target program zillions of times, trying to find input that causes it to crash.
It uses instrumentation of the target program's code to try to manipulate its input so that it explores as much of the target program as possible.
A roving cluster runs multiple copies of AFL on multiple machines, all fuzzing the same target. Roving's key contribution is to allow these machines to share and benefit from each other's work. If machine A finds an "interesting" test case that causes a new function to get invoked, machines B, C and D can all use this discovery to explore the rest of the program more efficiently.
A roving cluster consists of 1 server and N clients. Each client runs M copies of AFL (using AFL's existing parallelism settings), and uses the server to share their work with their peers. Each fuzzer on the client periodically (by default, every 5 mins) uploads to the server their current AFL state, including their queue. The server saves these states in memory.
Fuzzers take advantage of the work of their peers by downloading from the server the state of all clients in the cluster. They replace their current queue with the combined queues of all clients, and then continue fuzzing as before. This allows all clients to benefit from the new, interesting testcases that any individual client discovers.
This approach relies on the non-determinism of AFL. If every client deterministically ran the same test cases when given the same queue, we would simply be repeating the same work N times across N different clients. In reality, clients take the same queue and run in wildly different directions with it. This means that we cover more of the search space, faster.
That said, there is no formal partitioning of work, and there will be some amount of duplication of work between clients. We do not currently have any estimates of how much work is duplicated, but it is safe to say that running 10 roving clients will not get you 10x the edge-discovery rate of 1 client. Roving uses the same principle as AFL's own single-machine parallelism, so we still have good reason to believe that it is effective.
For now roving uses [Bazel][https://docs.bazel.build/versions/master/install.html] for its build. You'll need to download it in order to build roving.
AFLwith the path to afl, or make sure
- In the workdir, create a
- In the workdir, make a directory called
inputand populate it with a corpus
bazel build //cmd/srv
Once up, it will create a directory called
output that mirrors the
structure of the
output directory created by AFL. It will aggregate
crashes, hangs, and the queue.
There is also a basic (but improving!) admin page at
Clients should require almost no configuration.
bazel build //cmd/client
bazel-bin/cmd/client/darwin_amd64_stripped/client -- -server-hostport XYZ:123 -parallelism X
Clients will accumulate crashes and hangs in their working dir. They will sync them to the server.
Run the compiled binaries with the
-help flag or see the files in the
folder for advanced options.
The test suite is not particularly extensive, but you can run it using:
Roving clients should be very dumb
Roving clients should be very dumb and have very little configuration. This is so that clients can easily be brought up, pointed at any roving server of any type, and quickly start working.
If a roving server requires clients to be configured in a particular way (perhaps the server wants them to sync their work with it more frequently than normal), this should be passed as configuration to the server, which should then send it to the client when it starts up and joins the cluster.
Fuzzer-agnosticism is good but currently not essential
We would like roving to be fuzzer-agnostic in the future. It should be
possible to power your fuzzing using
any other reasonable fuzzer.
All of these fuzzers work in somewhat different ways and have somewhat
different structures and opinions. We are comfortable loosely coupling
afl for now - for example, we assume that fuzzer input and
output is structured in the way that
afl expects. However, we would like
multi-fuzzer support to be an achievable goal in the future, and would like
to avoid making decisions that would make this unreasonably difficult.
Running the examples
The example code bash scripts live in the
examples/c-serverto build the target and run the example server serving the C example target on the default port 1414
examples/generic-clientto run the example client
Your client should find a crash within 30 seconds.
examples/ruby-serverto run the example server serving the Ruby example target on the default port 1414
examples/generic-clientto run the example client
Your client should again find a crash within 30 seconds.
I asked some of my coworkers what they'd name a distributed fuzzy thing.
Evidently roving is extremely fuzzy, and winds up everywhere when you're working with it. Plus the testcases go roving and it's all very poetic.