From 6b1cc05ff786fc425fa723da1fe2e75338b0a3b7 Mon Sep 17 00:00:00 2001 From: Kevin Plybon Date: Thu, 5 Sep 2019 15:30:00 -0400 Subject: [PATCH 1/3] [Docs] Copy edit Index, Architecture, Getting Started. --- README.md | 6 +-- docs/clusterfuzz/architecture.md | 37 +++++++++-------- docs/getting-started/local_instance.md | 55 +++++++++++++++----------- docs/getting-started/prerequisites.md | 54 ++++++++++++------------- docs/index.md | 4 +- 5 files changed, 85 insertions(+), 71 deletions(-) diff --git a/README.md b/README.md index 7f8f55e50ff..d1d66cf79de 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,10 @@

-ClusterFuzz is a scalable fuzzing infrastructure which finds security and -stability issues in software. +ClusterFuzz is a scalable [fuzzing](https://en.wikipedia.org/wiki/Fuzzing) +infrastructure that finds security and stability issues in software. -It is used by Google for fuzzing the Chrome Browser, and serves as the fuzzing +Google uses ClusterFuzz to fuzz the Chrome Browser and as the fuzzing backend for [OSS-Fuzz]. ClusterFuzz provides many features which help seamlessly integrate fuzzing into diff --git a/docs/clusterfuzz/architecture.md b/docs/clusterfuzz/architecture.md index 29aa41cb1ab..3e3658b8222 100644 --- a/docs/clusterfuzz/architecture.md +++ b/docs/clusterfuzz/architecture.md @@ -7,10 +7,11 @@ parent: ClusterFuzz --- # Architecture +{: .no_toc} ![overview]({{ site.baseurl }}/images/overview.png) ClusterFuzz provides an automated end-to-end infrastructure for finding and -triaging crashes, minimizing reproducers, [bisecting], and verification of fixes. +triaging crashes, minimizing reproducers, [bisecting], and verifying fixes. - TOC {:toc} @@ -21,9 +22,9 @@ triaging crashes, minimizing reproducers, [bisecting], and verification of fixes ClusterFuzz is written in Python and Go. It runs on **Linux**, **macOS**, and **Windows**. ## Requirements -It runs on the [Google Cloud Platform](https://cloud.google.com/), and depends +ClusterFuzz runs on the [Google Cloud Platform](https://cloud.google.com/) and depends on a number of services: -- Compute Engine (Not strictly necessary. Bots can run anywhere). +- Compute Engine (Not strictly necessary. Bots can run anywhere.) - App Engine - Cloud Storage - Cloud Datastore @@ -31,25 +32,31 @@ on a number of services: - BigQuery - Stackdriver Logging and Monitoring -**Note**: The only bug tracker supported now is the Chromium hosted +**Note**: The only bug tracker we currently support is the Chromium-hosted [Monorail](https://opensource.google.com/projects/monorail). Support for custom -bug trackers will be added in the near future. +bug trackers will be added in the future. ### Local instance It's possible to run ClusterFuzz locally without these dependencies by using -local Google Cloud emulators, but some features which depend on BigQuery and +local Google Cloud emulators. If you do, some features that depend on BigQuery and Stackdriver will be disabled due to lack of emulator support. +**Note:** Local development is only supported on **Linux** and **macOS**. + ## Operation The two main components of ClusterFuzz are: - App Engine instance - A pool of [bots]({{ site.baseurl }}/reference/glossary/#bot) +### App Engine + The App Engine instance provides a web interface to access crashes, stats and other information. It's also responsible for scheduling regular cron jobs. -Bots are machines which run scheduled tasks. They lease tasks from platform +### Bots + +Bots are machines that run scheduled tasks. They lease tasks from platform specific queues. The main tasks that bots run are: - `fuzz`: Run a fuzzing session. - `progression`: Check if a testcase still reproduces or if it's fixed. @@ -59,16 +66,14 @@ specific queues. The main tasks that bots run are: }}/reference/glossary/#corpus) to smallest size based on coverage (libFuzzer only). - `analyze`: Run a manually uploaded testcase against a job to see if it crashes. -### Bots -There are 2 kinds of bots on ClusterFuzz - preemptible and non-preemptible. +There are two kinds of bots on ClusterFuzz: -Preemptible means that the machine can shutdown at any time. On these machines -we only run `fuzz` task. These machines are often cheaper on cloud providers, so -it's recommended to scale using these machines. +- **Preemptible**: The machine can shut down at any time, and can only run the +`fuzz` task. These machines are often cheaper on cloud +providers, so we recommended using them to scale. -Non-preemptible machines are not expected to shutdown. They are able to run all -tasks (including `fuzz`) and other critical tasks such as `progression` which -must run uninterrupted. +- **Non-preemptible**: The machine is not expected to shut down. It can run all +tasks, including critical ones like `progression` that must run uninterrupted. [bisecting]: https://en.wikipedia.org/wiki/Bisection_(software_engineering) -[minimization]: {{ site.baseurl }}/reference/glossary/#minimization +[minimization]: {{ site.baseurl }}/reference/glossary/#minimization \ No newline at end of file diff --git a/docs/getting-started/local_instance.md b/docs/getting-started/local_instance.md index 4c96b2d0647..967ceb1eaa0 100644 --- a/docs/getting-started/local_instance.md +++ b/docs/getting-started/local_instance.md @@ -7,8 +7,10 @@ permalink: /getting-started/local-instance/ --- # Local instance of ClusterFuzz +{: .no_toc} + You can run a local instance of ClusterFuzz to test core functionality. Note -that some features (e.g. [crash] and [fuzzer statistics]) are disabled to due to +that some features (like [crash] and [fuzzer statistics]) are disabled in local instances due to lack of Google Cloud emulators. - TOC @@ -21,6 +23,8 @@ lack of Google Cloud emulators. ## Running a local server +You can start a local server by running the following command: + ```bash # If you run the server for the first time or want to reset all data. python butler.py run_server --bootstrap @@ -29,48 +33,53 @@ python butler.py run_server --bootstrap python butler.py run_server ``` -This may take a few seconds to start. Once you see an output line like -`INFO admin_server.py:] Starting admin server`, you should be -able to navigate to [http://localhost:9000](http://localhost:9000) to view the -web interface. -Note: the local instance may use ports [other than 9000](https://github.com/google/clusterfuzz/blob/master/src/local/butler/constants.py), -such as 9008, for things like uploading files. Therefore, using the local -instance may break if the needed ports are unavailable or if you can only access -some of the needed ports from your browser (for example: because of port +It may take a few seconds to start. Once you see an output line like +`INFO admin_server.py:] Starting admin server`, you can see the web interface by navigating to [http://localhost:9000](http://localhost:9000). + +**Note:** The local instance may use ports [other than 9000](https://github.com/google/clusterfuzz/blob/master/src/local/butler/constants.py), +such as 9008, for things like uploading files. Your local +instance may break if the needed ports are unavailable, or if you can only access +some of the needed ports from your browser (for example, if you have port forwarding or firewall rules when accessing from another host). ## Running a local bot instance +You can run a ClusterFuzz bot in your local instance by running the following command: + ```bash python butler.py run_bot --name my-bot /path/to/my-bot # rename my-bot to anything ``` -This creates a copy of ClusterFuzz source under `/path/to/my-bot/clusterfuzz` -and runs the bot using it. Most of the bot artifacts like logs, fuzzers, -corpora, etc are created inside the `bot` subdirectory. +This creates a copy of the ClusterFuzz source under `/path/to/my-bot/clusterfuzz` +and uses it to run the bot. Most bot artifacts like logs, fuzzers, +and corpora are created inside the `bot` subdirectory. -If you plan to fuzz native GUI applications, it is advisable to run this command -in a virtual framebuffer (e.g. [Xvfb](https://en.wikipedia.org/wiki/Xvfb)). -Otherwise, you will see GUI dialogs while fuzzing and will be unable to use the -machine with ease. +If you plan to fuzz native GUI applications, we recommend you run this command +in a virtual framebuffer like [Xvfb](https://en.wikipedia.org/wiki/Xvfb). +Otherwise, you'll see GUI dialogs while fuzzing. ### Viewing logs + +You can see logs on your local instance by running the following command: + ```bash cd /path/to/my-bot/clusterfuzz/bot/logs tail -f bot.log ``` -Until you [set up the fuzzing jobs]({{ site.baseurl }}/setting-up-fuzzing/), -you will see a harmless error in the logs - `Failed to get any fuzzing tasks`. +**Note:** Until you [set up your fuzzing jobs]({{ site.baseurl }}/setting-up-fuzzing/), +you'll see a harmless error in the logs: `Failed to get any fuzzing tasks`. ## Local Google Cloud Storage We simulate [Google Cloud Storage] using your local filesystem. By default, this -is stored at `local/storage/local_gcs` in your ClusterFuzz checkout. You can -override it using `--storage-path` when running `run_server` command and then -specifying the same path using `--server-storage-path` when running `run_bot` +is stored at `local/storage/local_gcs` in your ClusterFuzz checkout. + +You can override the default location by doing the following: +1. Use the `--storage-path` flag with the `run_server` command. +2. Specify the same path using the `--server-storage-path` flag with the `run_bot` command. -Under this location, objects are stored in `/objects/` and +In the location you specify, objects are stored in `/objects/` and metadata is stored in `/metadata/`. -[Google Cloud Storage]: https://cloud.google.com/storage/ +[Google Cloud Storage]: https://cloud.google.com/storage/ \ No newline at end of file diff --git a/docs/getting-started/prerequisites.md b/docs/getting-started/prerequisites.md index 66c25d7e896..88895b4ad48 100644 --- a/docs/getting-started/prerequisites.md +++ b/docs/getting-started/prerequisites.md @@ -7,6 +7,9 @@ permalink: /getting-started/prerequisites/ --- # Prerequisites +{: .no_toc} + +This page explains how to set up your environment for using ClusterFuzz. - TOC {:toc} @@ -14,39 +17,39 @@ permalink: /getting-started/prerequisites/ --- ## Requirements Many features of ClusterFuzz depend on [Google Cloud -Platform](https://cloud.google.com) services (see -[this]({{ site.baseurl }}/architecture/#requirements) page for more details). -However, it's possible to run it locally without these dependencies for testing -purposes. +Platform](https://cloud.google.com) services, but it's possible to run it locally without these dependencies for testing purposes. See the [Architecture page]({{ site.baseurl }}/architecture/#requirements) for more details. -While ClusterFuzz runs on a number of platforms, local development is only -supported on **Linux** and **macOS**. +**Note:** Local development is only supported on **Linux** and **macOS**. ## Getting the code + +Clone the ClusterFuzz repository to your machine by running the following command: + ```bash git clone https://github.com/google/clusterfuzz cd clusterfuzz ``` -We recommend that you use the [last release version](https://github.com/google/clusterfuzz/releases/latest) -of our code for stability reasons (rather than master branch). -You can checkout a particular release using: +We recommend that you use the [latest release +version](https://github.com/google/clusterfuzz/releases/latest) of our code +(rather than master branch) for stability reasons. You can check out a +particular release using: ```bash git checkout tags/vX.Y.Z ``` -where X.Y.Z is the release version (e.g. 1.0.1). +where X.Y.Z is the release version (for example, 1.0.1). ## Installing prerequisites -### Google Cloud SDK -Install the Google Cloud SDK by following the instructions -[here](https://cloud.google.com/sdk/). +### Google Cloud SDK +Install the Google Cloud SDK by following the [online +instructions](https://cloud.google.com/sdk/). -### (Optional) Log in to your Google Cloud account -This is **not** necessary if you are simply running ClusterFuzz [locally]. +### Log in to your Google Cloud account +**Note:** This is **not** necessary if you're [running ClusterFuzz locally]. -If you are planning to set up ClusterFuzz in [production], you should +If you're planning to [set up ClusterFuzz in production], you should authenticate your account with the `gcloud` tool: ```bash @@ -54,20 +57,17 @@ gcloud auth application-default login gcloud auth login ``` -[production]: {{ "/production-setup/" | relative_url }} -[locally]: {{ "/getting-started/local-instance/" | relative_url }} +[set up ClusterFuzz in production]: {{ "/production-setup/" | relative_url }} +[running ClusterFuzz locally]: {{ "/getting-started/local-instance/" | relative_url }} ### Python programming language -Install Python 2.7. You can download it -[here](https://www.python.org/downloads/release/python-2715/). +[Download Python 2.7](https://www.python.org/downloads/release/python-2715/), then install it. If you already have Python installed, you can verify its version by running `python --version`. The minimum required version is 2.7.10. ### Go programming language -Install the Go programming language by following the instructions -[here](https://golang.org/doc/install). - +[Install the Go programming language](https://golang.org/doc/install). ### Other dependencies We provide a script for installing all other development dependencies on Linux @@ -87,14 +87,14 @@ local/install_deps.bash [homebrew]: https://brew.sh/ ## Loading virtualenv -Activate the virtualenv created by the `local/install_deps.bash` script. This -loads all the python dependencies in the current environment. +After you run the `local/install_deps.bash` script, activate the virtualenv by running the following command: ```bash source ENV/bin/activate ``` +This loads all the Python dependencies in the current environment. -Verify everything works by running: +You can verify that everything works by running: ```bash python butler.py --help -``` +``` \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index 48df7560df9..3ac0a4be6d6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -13,9 +13,9 @@ has_children: true

ClusterFuzz is a scalable [fuzzing](https://en.wikipedia.org/wiki/Fuzzing) -infrastructure which finds security and stability issues in software. +infrastructure that finds security and stability issues in software. -It is used by Google for fuzzing the Chrome Browser, and serves as the fuzzing +Google uses ClusterFuzz to fuzz the Chrome Browser and as the fuzzing backend for [OSS-Fuzz]. ClusterFuzz provides many features to seamlessly integrate fuzzing into From e3e0c83db2d93d5ade6536584b9a2fba5f87e70f Mon Sep 17 00:00:00 2001 From: Kevin Plybon Date: Thu, 5 Sep 2019 16:06:32 -0400 Subject: [PATCH 2/3] Minor tweaks implementing Max's feedback. --- docs/clusterfuzz/architecture.md | 8 ++++---- docs/getting-started/prerequisites.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/clusterfuzz/architecture.md b/docs/clusterfuzz/architecture.md index 3e3658b8222..b6a5a22410f 100644 --- a/docs/clusterfuzz/architecture.md +++ b/docs/clusterfuzz/architecture.md @@ -24,7 +24,7 @@ ClusterFuzz is written in Python and Go. It runs on **Linux**, **macOS**, and ** ## Requirements ClusterFuzz runs on the [Google Cloud Platform](https://cloud.google.com/) and depends on a number of services: -- Compute Engine (Not strictly necessary. Bots can run anywhere.) +- Compute Engine (Not strictly necessary. [Fuzzing bots](#bots) can run anywhere.) - App Engine - Cloud Storage - Cloud Datastore @@ -47,16 +47,16 @@ Stackdriver will be disabled due to lack of emulator support. The two main components of ClusterFuzz are: - App Engine instance -- A pool of [bots]({{ site.baseurl }}/reference/glossary/#bot) +- A pool of [fuzzing bots]({{ site.baseurl }}/reference/glossary/#bot) ### App Engine The App Engine instance provides a web interface to access crashes, stats and other information. It's also responsible for scheduling regular cron jobs. -### Bots +### Fuzzing Bots -Bots are machines that run scheduled tasks. They lease tasks from platform +Fuzzing bots are machines that run scheduled tasks. They lease tasks from platform specific queues. The main tasks that bots run are: - `fuzz`: Run a fuzzing session. - `progression`: Check if a testcase still reproduces or if it's fixed. diff --git a/docs/getting-started/prerequisites.md b/docs/getting-started/prerequisites.md index 88895b4ad48..537dc0df013 100644 --- a/docs/getting-started/prerequisites.md +++ b/docs/getting-started/prerequisites.md @@ -42,7 +42,7 @@ where X.Y.Z is the release version (for example, 1.0.1). ## Installing prerequisites -### Google Cloud SDK +### Google Cloud SDK Install the Google Cloud SDK by following the [online instructions](https://cloud.google.com/sdk/). From 7d046b7696b26a71d085ab9f97c0487ecb0796a9 Mon Sep 17 00:00:00 2001 From: Kevin Plybon Date: Thu, 5 Sep 2019 16:24:39 -0400 Subject: [PATCH 3/3] Minor tweak for Jonathan's comments. --- docs/clusterfuzz/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/clusterfuzz/architecture.md b/docs/clusterfuzz/architecture.md index b6a5a22410f..9888f113f99 100644 --- a/docs/clusterfuzz/architecture.md +++ b/docs/clusterfuzz/architecture.md @@ -41,7 +41,7 @@ It's possible to run ClusterFuzz locally without these dependencies by using local Google Cloud emulators. If you do, some features that depend on BigQuery and Stackdriver will be disabled due to lack of emulator support. -**Note:** Local development is only supported on **Linux** and **macOS**. +**Note:** Local instances are only supported on **Linux** and **macOS**. ## Operation The two main components of ClusterFuzz are: