Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
<img src="docs/images/logo.png" width="400">
</p>

ClusterFuzz is a scalable fuzzing infrastructure which finds security and
stability issues in software.
ClusterFuzz is a scalable [fuzzing](https://en.wikipedia.org/wiki/Fuzzing)
infrastructure that finds security and stability issues in software.

It is used by Google for fuzzing the Chrome Browser, and serves as the fuzzing
Google uses ClusterFuzz to fuzz the Chrome Browser and as the fuzzing
backend for [OSS-Fuzz].

ClusterFuzz provides many features which help seamlessly integrate fuzzing into
Expand Down
39 changes: 22 additions & 17 deletions docs/clusterfuzz/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@ parent: ClusterFuzz
---

# Architecture
{: .no_toc}
![overview]({{ site.baseurl }}/images/overview.png)

ClusterFuzz provides an automated end-to-end infrastructure for finding and
triaging crashes, minimizing reproducers, [bisecting], and verification of fixes.
triaging crashes, minimizing reproducers, [bisecting], and verifying fixes.

- TOC
{:toc}
Expand All @@ -21,35 +22,41 @@ triaging crashes, minimizing reproducers, [bisecting], and verification of fixes
ClusterFuzz is written in Python and Go. It runs on **Linux**, **macOS**, and **Windows**.

## Requirements
It runs on the [Google Cloud Platform](https://cloud.google.com/), and depends
ClusterFuzz runs on the [Google Cloud Platform](https://cloud.google.com/) and depends
on a number of services:
- Compute Engine (Not strictly necessary. Bots can run anywhere).
- Compute Engine (Not strictly necessary. [Fuzzing bots](#bots) can run anywhere.)
- App Engine
- Cloud Storage
- Cloud Datastore
- Cloud Pub/Sub
- BigQuery
- Stackdriver Logging and Monitoring

**Note**: The only bug tracker supported now is the Chromium hosted
**Note**: The only bug tracker we currently support is the Chromium-hosted
[Monorail](https://opensource.google.com/projects/monorail). Support for custom
bug trackers will be added in the near future.
bug trackers will be added in the future.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Kevin, please ignore) @oliverchang do you want to re-write this sentence to mention bug tracker extendability?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, added to my TODO.


### Local instance
It's possible to run ClusterFuzz locally without these dependencies by using
local Google Cloud emulators, but some features which depend on BigQuery and
local Google Cloud emulators. If you do, some features that depend on BigQuery and
Stackdriver will be disabled due to lack of emulator support.

**Note:** Local instances are only supported on **Linux** and **macOS**.

## Operation
The two main components of ClusterFuzz are:

- App Engine instance
- A pool of [bots]({{ site.baseurl }}/reference/glossary/#bot)
- A pool of [fuzzing bots]({{ site.baseurl }}/reference/glossary/#bot)

### App Engine

The App Engine instance provides a web interface to access crashes, stats and
other information. It's also responsible for scheduling regular cron jobs.

Bots are machines which run scheduled tasks. They lease tasks from platform
### Fuzzing Bots

Fuzzing bots are machines that run scheduled tasks. They lease tasks from platform
specific queues. The main tasks that bots run are:
- `fuzz`: Run a fuzzing session.
- `progression`: Check if a testcase still reproduces or if it's fixed.
Expand All @@ -59,16 +66,14 @@ specific queues. The main tasks that bots run are:
}}/reference/glossary/#corpus) to smallest size based on coverage (libFuzzer only).
- `analyze`: Run a manually uploaded testcase against a job to see if it crashes.

### Bots
There are 2 kinds of bots on ClusterFuzz - preemptible and non-preemptible.
There are two kinds of bots on ClusterFuzz:

Preemptible means that the machine can shutdown at any time. On these machines
we only run `fuzz` task. These machines are often cheaper on cloud providers, so
it's recommended to scale using these machines.
- **Preemptible**: The machine can shut down at any time, and can only run the
`fuzz` task. These machines are often cheaper on cloud
providers, so we recommended using them to scale.

Non-preemptible machines are not expected to shutdown. They are able to run all
tasks (including `fuzz`) and other critical tasks such as `progression` which
must run uninterrupted.
- **Non-preemptible**: The machine is not expected to shut down. It can run all
tasks, including critical ones like `progression` that must run uninterrupted.

[bisecting]: https://en.wikipedia.org/wiki/Bisection_(software_engineering)
[minimization]: {{ site.baseurl }}/reference/glossary/#minimization
[minimization]: {{ site.baseurl }}/reference/glossary/#minimization
55 changes: 32 additions & 23 deletions docs/getting-started/local_instance.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@ permalink: /getting-started/local-instance/
---

# Local instance of ClusterFuzz
{: .no_toc}

You can run a local instance of ClusterFuzz to test core functionality. Note
that some features (e.g. [crash] and [fuzzer statistics]) are disabled to due to
that some features (like [crash] and [fuzzer statistics]) are disabled in local instances due to
lack of Google Cloud emulators.

- TOC
Expand All @@ -21,6 +23,8 @@ lack of Google Cloud emulators.

## Running a local server

You can start a local server by running the following command:

```bash
# If you run the server for the first time or want to reset all data.
python butler.py run_server --bootstrap
Expand All @@ -29,48 +33,53 @@ python butler.py run_server --bootstrap
python butler.py run_server
```

This may take a few seconds to start. Once you see an output line like
`INFO <timestamp> admin_server.py:<num>] Starting admin server`, you should be
able to navigate to [http://localhost:9000](http://localhost:9000) to view the
web interface.
Note: the local instance may use ports [other than 9000](https://github.com/google/clusterfuzz/blob/master/src/local/butler/constants.py),
such as 9008, for things like uploading files. Therefore, using the local
instance may break if the needed ports are unavailable or if you can only access
some of the needed ports from your browser (for example: because of port
It may take a few seconds to start. Once you see an output line like
`INFO <timestamp> admin_server.py:<num>] Starting admin server`, you can see the web interface by navigating to [http://localhost:9000](http://localhost:9000).

**Note:** The local instance may use ports [other than 9000](https://github.com/google/clusterfuzz/blob/master/src/local/butler/constants.py),
such as 9008, for things like uploading files. Your local
instance may break if the needed ports are unavailable, or if you can only access
some of the needed ports from your browser (for example, if you have port
forwarding or firewall rules when accessing from another host).

## Running a local bot instance

You can run a ClusterFuzz bot in your local instance by running the following command:

```bash
python butler.py run_bot --name my-bot /path/to/my-bot # rename my-bot to anything
```

This creates a copy of ClusterFuzz source under `/path/to/my-bot/clusterfuzz`
and runs the bot using it. Most of the bot artifacts like logs, fuzzers,
corpora, etc are created inside the `bot` subdirectory.
This creates a copy of the ClusterFuzz source under `/path/to/my-bot/clusterfuzz`
and uses it to run the bot. Most bot artifacts like logs, fuzzers,
and corpora are created inside the `bot` subdirectory.

If you plan to fuzz native GUI applications, it is advisable to run this command
in a virtual framebuffer (e.g. [Xvfb](https://en.wikipedia.org/wiki/Xvfb)).
Otherwise, you will see GUI dialogs while fuzzing and will be unable to use the
machine with ease.
If you plan to fuzz native GUI applications, we recommend you run this command
in a virtual framebuffer like [Xvfb](https://en.wikipedia.org/wiki/Xvfb).
Otherwise, you'll see GUI dialogs while fuzzing.

### Viewing logs

You can see logs on your local instance by running the following command:

```bash
cd /path/to/my-bot/clusterfuzz/bot/logs
tail -f bot.log
```

Until you [set up the fuzzing jobs]({{ site.baseurl }}/setting-up-fuzzing/),
you will see a harmless error in the logs - `Failed to get any fuzzing tasks`.
**Note:** Until you [set up your fuzzing jobs]({{ site.baseurl }}/setting-up-fuzzing/),
you'll see a harmless error in the logs: `Failed to get any fuzzing tasks`.

## Local Google Cloud Storage
We simulate [Google Cloud Storage] using your local filesystem. By default, this
is stored at `local/storage/local_gcs` in your ClusterFuzz checkout. You can
override it using `--storage-path` when running `run_server` command and then
specifying the same path using `--server-storage-path` when running `run_bot`
is stored at `local/storage/local_gcs` in your ClusterFuzz checkout.

You can override the default location by doing the following:
1. Use the `--storage-path` flag with the `run_server` command.
2. Specify the same path using the `--server-storage-path` flag with the `run_bot`
command.

Under this location, objects are stored in `<bucket>/objects/<object path>` and
In the location you specify, objects are stored in `<bucket>/objects/<object path>` and
metadata is stored in `<bucket>/metadata/<object path>`.

[Google Cloud Storage]: https://cloud.google.com/storage/
[Google Cloud Storage]: https://cloud.google.com/storage/
52 changes: 26 additions & 26 deletions docs/getting-started/prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,67 +7,67 @@ permalink: /getting-started/prerequisites/
---

# Prerequisites
{: .no_toc}

This page explains how to set up your environment for using ClusterFuzz.

- TOC
{:toc}

---
## Requirements
Many features of ClusterFuzz depend on [Google Cloud
Platform](https://cloud.google.com) services (see
[this]({{ site.baseurl }}/architecture/#requirements) page for more details).
However, it's possible to run it locally without these dependencies for testing
purposes.
Platform](https://cloud.google.com) services, but it's possible to run it locally without these dependencies for testing purposes. See the [Architecture page]({{ site.baseurl }}/architecture/#requirements) for more details.

While ClusterFuzz runs on a number of platforms, local development is only
supported on **Linux** and **macOS**.
**Note:** Local development is only supported on **Linux** and **macOS**.

## Getting the code

Clone the ClusterFuzz repository to your machine by running the following command:

```bash
git clone https://github.com/google/clusterfuzz
cd clusterfuzz
```

We recommend that you use the [last release version](https://github.com/google/clusterfuzz/releases/latest)
of our code for stability reasons (rather than master branch).
You can checkout a particular release using:
We recommend that you use the [latest release
version](https://github.com/google/clusterfuzz/releases/latest) of our code
(rather than master branch) for stability reasons. You can check out a
particular release using:

```bash
git checkout tags/vX.Y.Z
```
where X.Y.Z is the release version (e.g. 1.0.1).
where X.Y.Z is the release version (for example, 1.0.1).

## Installing prerequisites

### Google Cloud SDK
Install the Google Cloud SDK by following the instructions
[here](https://cloud.google.com/sdk/).
Install the Google Cloud SDK by following the [online
instructions](https://cloud.google.com/sdk/).

### (Optional) Log in to your Google Cloud account
This is **not** necessary if you are simply running ClusterFuzz [locally].
### Log in to your Google Cloud account
**Note:** This is **not** necessary if you're [running ClusterFuzz locally].

If you are planning to set up ClusterFuzz in [production], you should
If you're planning to [set up ClusterFuzz in production], you should
authenticate your account with the `gcloud` tool:

```bash
gcloud auth application-default login
gcloud auth login
```

[production]: {{ "/production-setup/" | relative_url }}
[locally]: {{ "/getting-started/local-instance/" | relative_url }}
[set up ClusterFuzz in production]: {{ "/production-setup/" | relative_url }}
[running ClusterFuzz locally]: {{ "/getting-started/local-instance/" | relative_url }}

### Python programming language
Install Python 2.7. You can download it
[here](https://www.python.org/downloads/release/python-2715/).
[Download Python 2.7](https://www.python.org/downloads/release/python-2715/), then install it.

If you already have Python installed, you can verify its version by running `python --version`.
The minimum required version is 2.7.10.

### Go programming language
Install the Go programming language by following the instructions
[here](https://golang.org/doc/install).

[Install the Go programming language](https://golang.org/doc/install).

### Other dependencies
We provide a script for installing all other development dependencies on Linux
Expand All @@ -87,14 +87,14 @@ local/install_deps.bash
[homebrew]: https://brew.sh/

## Loading virtualenv
Activate the virtualenv created by the `local/install_deps.bash` script. This
loads all the python dependencies in the current environment.
After you run the `local/install_deps.bash` script, activate the virtualenv by running the following command:

```bash
source ENV/bin/activate
```
This loads all the Python dependencies in the current environment.

Verify everything works by running:
You can verify that everything works by running:
```bash
python butler.py --help
```
```
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ has_children: true
</p>

ClusterFuzz is a scalable [fuzzing](https://en.wikipedia.org/wiki/Fuzzing)
infrastructure which finds security and stability issues in software.
infrastructure that finds security and stability issues in software.

It is used by Google for fuzzing the Chrome Browser, and serves as the fuzzing
Google uses ClusterFuzz to fuzz the Chrome Browser and as the fuzzing
backend for [OSS-Fuzz].

ClusterFuzz provides many features to seamlessly integrate fuzzing into
Expand Down