Skip to content


Browse files Browse the repository at this point in the history
Add an option to submit test-perf result to perfherder
  • Loading branch information
shinglyu committed Dec 1, 2016
1 parent 16199b4 commit ddd0322
Show file tree
Hide file tree
Showing 4 changed files with 38 additions and 31 deletions.
36 changes: 11 additions & 25 deletions etc/ci/performance/
Expand Up @@ -7,27 +7,21 @@ Servo Page Load Time Test

# Basic Usage

## Prepare the test runner
`./mach test-perf` can be used to run a performance test on your servo build. The test result JSON will be saved to `etc/ci/performance/output/`. You can then run `python` to compare these two test results. Run `python -h` for instructions.

# Setup for CI machine
## CI for Servo

* Clone this repo
* Download [](, extract it to `page_load_test/tp5n`
* Run `` to transform the tp5n manifest to our format
* Install the Python3 `treeherder-client` package. For example, to install it in a virtualenv: `python3 -m virtualenv venv; source venv/bin/activate; pip install "treeherder-client>=3.0.0"`
* Setup your Treeherder client ID and secret as environment variables `TREEHERDER_CLIENT_ID` and `TREEHERDER_CLIENT_SECRET`
* Run `./mach test-perf --submit` to run and submit the result to Perfherder.

## Build Servo
* Clone the servo repo
* Compile release build
* Run `` in the servo repo, save the output as `revision.json`
* Put your `servo` binary, `revision.json` and `resources` folder in `etc/ci/performance/servo/`
## CI for Gecko

## Run
* Activate the virutalenv: `source venv/bin/activate`
* Sync your system clock before running, the Perfherder API SSL check will fail if your system clock is not accurate. (e.g. `sudo nptdate`)
* Run ` [--servo|--gecko] [--submit]`
- choose `servo` or `gecko` as the testing engine
- enable `submit`, if you want to submit to perfherder
* Test results are submitted to
* Install Firefox Nightly in your PATH
* Download [geckodriver]( and add it to the `PATH`
* `pip install selenium`
* Run `python` to test
* Run ` --gecko --submit`

# How it works

Expand Down Expand Up @@ -61,14 +55,6 @@ If you want to test the data submission code in `` withou
* `./ create_credentials <username> <email> "description"`, the email has to match your logged in user. Remember to log-in through the Web UI once before you run this.
* Setup your Treeherder client ID and secret as environment variables `TREEHERDER_CLIENT_ID` and `TREEHERDER_CLIENT_SECRET`

## For Gecko

* Install Firefox Nightly in your PATH
* Download [geckodriver]( and add it to the `PATH`
* `pip install selenium`
* Run `python` to test

# Troubleshooting

If you saw this error message:
Expand Down
2 changes: 1 addition & 1 deletion etc/ci/performance/
Expand Up @@ -57,7 +57,7 @@ then
# results appear on the same date. Use the correct result when Perfherder
# allows us to change the date.
python3 \
"${output:-}" "${engine}" "${PERF_FILE}" servo/revision.json
"${engine}" "${PERF_FILE}" servo/revision.json

echo "Stopping the local server"
Expand Down
16 changes: 13 additions & 3 deletions etc/ci/performance/
Expand Up @@ -32,6 +32,16 @@ PS1="" source venv/bin/activate
pip install "treeherder-client>=3.0.0"

mkdir -p servo
mkdir -p output
./ > servo/revision.json && \
./ --servo
mkdir -p output # Test result will be saved to output/perf-<timestamp>.json
./ > servo/revision.json

if [[ "${#}" -eq 1 ]]; then
if [[ "${1}" = "--submit" ]]; then
./ --servo --submit
echo "Unrecognized argument: ${1}; Ignore and proceed without submitting"
./ --servo
./ --servo
15 changes: 13 additions & 2 deletions python/servo/
Expand Up @@ -166,12 +166,23 @@ def suite_for_path(self, path_arg):
description='Run the page load performance test',
def test_perf(self):
@CommandArgument('--submit', default=False, action="store_true",
help="submit the data to perfherder")
def test_perf(self, submit=False):

env = self.build_env()
return call(["bash", ""],
cmd = ["bash", ""]
if submit:
if not ("TREEHERDER_CLIENT_ID" in os.environ and
print("Please set the environment variable \"TREEHERDER_CLIENT_ID\""
" and \"TREEHERDER_CLIENT_SECRET\" to submit the performance"
" test result to perfherder")
return 1
cmd += ["--submit"]
return call(cmd,
cwd=path.join("etc", "ci", "performance"))

Expand Down

0 comments on commit ddd0322

Please sign in to comment.