Benchmarking Dotty
bin/master
: for benchmarking the masterbin/run
: for manually benchmarking a specific period (after 2017.10.27)bin/pull
: for testing a specific pull request
Check the documentation in the source code of each command for more detail.
Update the profiles below:
profiles/default.yml
profiles/ci.yml
Then run bin/sync-profiles
to update the test files and json files.
Put regression jobs as scripts under jobs/
.
bin/gauge 6514 49f254a
The command bin/gauge
accepts the following options:
-o File
: output file for test data, defaults toFILE
inbin/config
-p PLAN
: the test plan to execute, defaults toPLAN
inbin/config
To only measure a specific test target, just create a test plan for that test.
The command bin/run
allows schedule history benchmarks for a designated test target.
It accepts the following options:
-p PLAN
: the test plan to execute-f PR
: from which PR (excluded)-t PR
: to which PR (excluded)-s STEP
: execute test plan every n pull requests, defaults to 0-o FILE
: output file for test data, defaults toFILE
inbin/config
The command only schedules the tasks under the directory jobs/
, which will be executed
by the scheduler.
- Put new test in relevant profiles under
profiles/*.yml
- Make a pull request
cp profiles/default.plan profiles/scalapb.plan
# edit profiles/scalapb.plan to only keep the test relevant for debug
# run test with a PR and commit hash -- PR can be a random number
bin/gauge -p profiles/scalapb.plan 6179 54fb992
Current kernel: 4.4.0-78-generic #99-Ubuntu
- Install all dependencies listed above
- Disable system update to use a fixed version of kernel
# edit file /etc/apt/apt.conf.d/10periodic
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";
- Clone current project, and under the project root:
git clone git@github.com:lampepfl/dotty.git
git clone git@github.com:liufengyun/bench-web.git web
curl -Lo bin/coursier https://git.io/coursier-cli && chmod +x bin/coursier && bin/coursier --help
- Update configuration in
bin/config
- Authorize
ghi config --auth
, make sure it works from command line - Make sure
git
can push to current project - Setup cron jobs listed in
bin/cron
- Checkout the branch
gh-pages
of current project as a foldersite
under current project - Make a git repo under the directory
data
for storing test data, make suregit push origin master:master
works for the repo. - In
dotty
folder, rungit remote add staging https://github.com/dotty-staging/dotty.git
- run
bin/synch-profiles
once - Run static web server for logs:
ecstatic ./logs --port 8000 --content-type text/plain
- run
bin/cpu
once - start
bin/poll
andbin/queue
as long-running service