Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Bug 10452 - Page Load Time Test Runner
- Loading branch information
Showing
15 changed files
with
1,485 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
servo/* | ||
output.png | ||
output/* | ||
|
||
__pycache__/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
Servo Page Load Time Test | ||
============== | ||
|
||
# Prerequisites | ||
|
||
* Python3 | ||
|
||
# Basic Usage | ||
|
||
## Prepare the test runner | ||
|
||
* Clone this repo | ||
* Download [tp5n.zip](http://people.mozilla.org/~jmaher/taloszips/zips/tp5n.zip), extract it to `page_load_test/tp5n` | ||
* Run `prepare_manifest.sh` to transform the tp5n manifest to our format | ||
* Install the Python3 `treeherder-client` package. For example, to install it in a virtualenv: `python3 -m virtualenv venv; source venv/bin/activate; pip install treeherder-client` | ||
* Setup your Treeherder client ID and secret as environment variables `TREEHERDER_CLIENT_ID` and `TREEHERDER_CLIENT_SECRET` | ||
|
||
## Build Servo | ||
* Clone the servo repo | ||
* Compile release build | ||
* Run `git_log_to_json.sh` in the servo repo, save the output as `revision.json` | ||
* Put your `servo` binary, `revision.json` and `resources` folder in `etc/ci/performance/servo/` | ||
|
||
## Run | ||
* Activate the virutalenv: `source venv/bin/activate` | ||
* Sync your system clock before running, the Perfherder API SSL check will fail if your system clock is not accurate. (e.g. `sudo nptdate tw.pool.ntp.org`) | ||
* Run `test_all.sh` | ||
* Test results are submitted to https://treeherder.mozilla.org/#/jobs?repo=servo | ||
|
||
# How it works | ||
|
||
* The testcase is from tp5, every testcase will run 20 times, and we take the median. | ||
* Some of the tests will make Servo run forever, it's disabled right now. See https://github.com/servo/servo/issues/11087 | ||
* Each testcase is a subtest on Perfherder, and their summary time is the geometric mean of all the subtests. | ||
* Notice that the test is different from the Talos TP5 test we run for Gecko. So you can NOT conclude that Servo is "faster" or "slower" than Gecko from this test. | ||
|
||
# Unit tests | ||
|
||
You can run all unit tests (include 3rd-party libraries) with `python -m pytest`. | ||
|
||
Individual test can be run by `python -m pytest <filename>`: | ||
|
||
* `test_runner.py` | ||
* `test_submit_to_perfherder.py` | ||
|
||
# Advanced Usage | ||
|
||
## Test Perfherder Locally | ||
|
||
If you want to test the data submission code in `submit_to_perfherder.py` without getting a credential for the production server, you can setup a local treeherder VM. If you don't need to test `submit_to_perfherder.py`, you can skip this step. | ||
|
||
* Add `192.168.33.10 local.treeherder.mozilla.org` to `/etc/hosts` | ||
* `git clone https://github.com/mozilla/treeherder; cd treeherder` | ||
* `vagrant up` | ||
* `vagrant ssh` | ||
* `./bin/run_gunicorn` | ||
* Outside of vm, open `http://local.treeherder.mozilla.org` and login to create an account | ||
* `vagrant ssh` | ||
* `./manage.py create_credentials <username> <email> "description"`, the email has to match your logged in user. Remember to log-in through the Web UI once before you run this. | ||
* Setup your Treeherder client ID and secret as environment variables `TREEHERDER_CLIENT_ID` and `TREEHERDER_CLIENT_SECRET` | ||
|
||
## For Gecko | ||
|
||
* Install Firefox Nightly in your PATH | ||
* Install [jpm](https://developer.mozilla.org/en-US/Add-ons/SDK/Tools/jpm#Installation) | ||
* Run `jpm xpi` in the `firefox/addon` folder | ||
* Install the generated `xpi` file to your Firefox Nightly | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
#Servo Performance Comparison | ||
Monitor website rendering performance |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../../../user-agent-js/01.perf-timing.js |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
var self = require("sdk/self"); | ||
var pageMod = require("sdk/page-mod"); | ||
|
||
pageMod.PageMod({ | ||
include: "*", | ||
contentScriptFile: self.data.url('perf.js'), | ||
attachTo: ["top", "existing"] | ||
}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
{ | ||
"title": "Servo Performance Comparison", | ||
"name": "addon", | ||
"version": "0.0.1", | ||
"description": "Monitor website rendering performance", | ||
"main": "index.js", | ||
"author": "The Servo team", | ||
"engines": { | ||
"firefox": ">=38.0a1", | ||
"fennec": ">=38.0a1" | ||
}, | ||
"license": "MPL", | ||
"keywords": [ | ||
"jetpack" | ||
] | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
#!/usr/bin/env bash | ||
|
||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
|
||
set -o errexit | ||
set -o nounset | ||
set -o pipefail | ||
|
||
# Don't include body to prevent multiline and unescaped body string | ||
git log -n 1 --pretty=format:'{%n "commit": "%H",%n "subject": "%s",%n | ||
"author": {%n "name": "%aN",%n "email": "%aE",%n | ||
"timestamp": "%at"%n }%n %n}' | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
http://localhost:8000/tp6/news.ycombinator.com/index.html | ||
# http://localhost:8000/page_load_test/163.com/www.163.com/index.html | ||
http://localhost:8000/page_load_test/56.com/www.56.com/index.html | ||
# http://localhost:8000/page_load_test/aljazeera.net/aljazeera.net/portal.html | ||
http://localhost:8000/page_load_test/amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/507846.html | ||
http://localhost:8000/page_load_test/bbc.co.uk/www.bbc.co.uk/news/index.html | ||
http://localhost:8000/page_load_test/beatonna.livejournal.com/beatonna.livejournal.com/index.html | ||
# http://localhost:8000/page_load_test/bild.de/www.bild.de/index.html | ||
http://localhost:8000/page_load_test/cgi.ebay.com/cgi.ebay.com/ALL-NEW-KINDLE-3-eBOOK-WIRELESS-READING-DEVICE-W-WIFI-/130496077314@pt=LH_DefaultDomain_0&hash=item1e622c1e02.html | ||
http://localhost:8000/page_load_test/chemistry.about.com/chemistry.about.com/index.html | ||
# http://localhost:8000/page_load_test/chinaz.com/chinaz.com/index.html | ||
http://localhost:8000/page_load_test/cnn.com/www.cnn.com/index.html | ||
http://localhost:8000/page_load_test/dailymail.co.uk/www.dailymail.co.uk/ushome/index.html | ||
http://localhost:8000/page_load_test/dailymotion.com/www.dailymotion.com/us.html | ||
# http://localhost:8000/page_load_test/digg.com/digg.com/news/story/New_logo_for_Mozilla_Firefox_browser.html | ||
http://localhost:8000/page_load_test/ezinearticles.com/ezinearticles.com/index.html@Migraine-Ocular---The-Eye-Migraines&id=4684133.html | ||
http://localhost:8000/page_load_test/globo.com/www.globo.com/index.html | ||
http://localhost:8000/page_load_test/google.com/www.google.com/search@q=mozilla.html | ||
http://localhost:8000/page_load_test/goo.ne.jp/goo.ne.jp/index.html | ||
# http://localhost:8000/page_load_test/guardian.co.uk/www.guardian.co.uk/index.html | ||
# http://localhost:8000/page_load_test/homeway.com.cn/www.hexun.com/index.html | ||
http://localhost:8000/page_load_test/huffingtonpost.com/www.huffingtonpost.com/index.html | ||
# http://localhost:8000/page_load_test/ifeng.com/ifeng.com/index.html | ||
# http://localhost:8000/page_load_test/imdb.com/www.imdb.com/title/tt1099212/index.html | ||
http://localhost:8000/page_load_test/imgur.com/imgur.com/gallery/index.html | ||
# http://localhost:8000/page_load_test/indiatimes.com/www.indiatimes.com/index.html | ||
http://localhost:8000/page_load_test/mail.ru/mail.ru/index.html | ||
# http://localhost:8000/page_load_test/mashable.com/mashable.com/index.html | ||
http://localhost:8000/page_load_test/media.photobucket.com/media.photobucket.com/image/funny%20gif/findstuff22/Best%20Images/Funny/funny-gif1.jpg@o=1.html | ||
http://localhost:8000/page_load_test/myspace.com/www.myspace.com/albumart.html | ||
# http://localhost:8000/page_load_test/naver.com/www.naver.com/index.html | ||
# http://localhost:8000/page_load_test/noimpactman.typepad.com/noimpactman.typepad.com/index.html | ||
http://localhost:8000/page_load_test/page.renren.com/page.renren.com/index.html | ||
# http://localhost:8000/page_load_test/people.com.cn/people.com.cn/index.html | ||
http://localhost:8000/page_load_test/rakuten.co.jp/www.rakuten.co.jp/index.html | ||
http://localhost:8000/page_load_test/reddit.com/www.reddit.com/index.html | ||
# http://localhost:8000/page_load_test/reuters.com/www.reuters.com/index.html | ||
# http://localhost:8000/page_load_test/slideshare.net/www.slideshare.net/jameswillamor/lolcats-in-popular-culture-a-historical-perspective.html | ||
# http://localhost:8000/page_load_test/sohu.com/www.sohu.com/index.html | ||
http://localhost:8000/page_load_test/spiegel.de/www.spiegel.de/index.html | ||
http://localhost:8000/page_load_test/stackoverflow.com/stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered.html | ||
# http://localhost:8000/page_load_test/store.apple.com/store.apple.com/us@mco=Nzc1MjMwNA.html | ||
# http://localhost:8000/page_load_test/thepiratebay.org/thepiratebay.org/top/201.html | ||
http://localhost:8000/page_load_test/tudou.com/www.tudou.com/index.html | ||
# http://localhost:8000/page_load_test/uol.com.br/www.uol.com.br/index.html | ||
http://localhost:8000/page_load_test/w3.org/www.w3.org/standards/webdesign/htmlcss.html | ||
# http://localhost:8000/page_load_test/wsj.com/online.wsj.com/home-page.html | ||
# http://localhost:8000/page_load_test/xinhuanet.com/xinhuanet.com/index.html | ||
http://localhost:8000/page_load_test/xunlei.com/xunlei.com/index.html | ||
http://localhost:8000/page_load_test/yelp.com/www.yelp.com/biz/alexanders-steakhouse-cupertino.html | ||
http://localhost:8000/page_load_test/youku.com/www.youku.com/index.html | ||
http://localhost:8000/page_load_test/youtube.com/www.youtube.com/music.html |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
#!/usr/bin/env bash | ||
|
||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
|
||
set -o errexit | ||
set -o nounset | ||
set -o pipefail | ||
|
||
# TP5 manifest uses `localhost`, but our local server probably don't use port 80 | ||
sed 's/localhost\/page_load_test\/tp5n/localhost:8000\/page_load_test\/tp5n/g' \ | ||
./page_load_test/tp5n/tp5o.manifest > ./page_load_test/tp5n/tp5o_8000.manifest | ||
|
Oops, something went wrong.