Skip to content

Commit

Permalink
Update perf try job documentation
Browse files Browse the repository at this point in the history
Updating documentation after Pinpoint gave users the ability to control:
- both base and exp arm
- patches on both arms
- pairwise testing
- attempt count

Bug: chromium:1408371
Change-Id: I116ab0cd9252034acdb8096ded1599cab8a80f2a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4199211
Reviewed-by: Sean McCullough <seanmccullough@google.com>
Commit-Queue: Leina Sun <sunxiaodi@google.com>
Cr-Commit-Position: refs/heads/main@{#1098198}
  • Loading branch information
Leina Sun authored and Chromium LUCI CQ committed Jan 28, 2023
1 parent 46907e2 commit f31a3e2
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 3 deletions.
Binary file modified docs/speed/images/pinpoint-perf-try-dialog.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
13 changes: 10 additions & 3 deletions docs/speed/perf_trybots.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ patch applied.
automatically runs multiple times and aggregates the results.
* Some regressions reproduce on some devices but not others, and Pinpoint will
run the job on multiple devices.
* Each iteration runs both arms on the same device, eliminating confounding factors like across-device variability

## Starting a perf try job

Expand All @@ -39,7 +40,6 @@ Bot | The device type to run the test on. All hardware configurations in our per
Benchmark | A telemetry benchmark. E.g. `system_health.common_desktop`<br><br>All the telemetry benchmarks are supported by the perf trybots. To get a full list, run `tools/perf/run_benchmark list`<br><br>To learn more about the benchmarks, you can read about the [system health benchmarks](https://docs.google.com/document/d/1BM_6lBrPzpMNMtcyi2NFKGIzmzIQ1oH3OlNG27kDGNU/edit?ts=57e92782), which test Chrome's performance at a high level, and the [benchmark harnesses](https://docs.google.com/spreadsheets/d/1ZdQ9OHqEjF5v8dqNjd7lGUjJnK6sgi8MiqO7eZVMgD0/edit#gid=0), which cover more specific areas.
Story | (optional) A specific story from the benchmark to run. Note that if the story you want isn't on the dropdown it could be because the story is new and so the Chromeperf dashboard database doesn't know about it yet. In that case you can still free-form type the exact story name into the field.
Story Tags | (optional) A list of story tags. All stories in the given benchmark that match any of the tags will be run.
Extra Test Arguments | (optional) Extra arguments for the test. E.g. `--extra-chrome-categories="foo,bar"`<br><br>To see all arguments, run `tools/perf/run_benchmark run --help`

Note that you must provide either a Story or a Story Tag for Pinpoint to run.
Per [this explanation](https://bugs.chromium.org/p/chromium/issues/detail?id=1017811#c6), running an entire benchmark on Pinpoint can cause significant problems if the benchmark is large. For this reason, some small benchmarks have an 'all' tag available that applies to all the stories in the benchmark, so please use that tag to run all the stories for a small benchmark. Please see [this bug](https://bugs.chromium.org/p/chromium/issues/detail?id=1023451) for details on work to add the 'all' tag to more benchmarks. If you want to run a large benchmark, consider choosing one of the tags that benchmark provides to select a subset of the available stories for that benchmark.
Expand All @@ -48,9 +48,16 @@ Per [this explanation](https://bugs.chromium.org/p/chromium/issues/detail?id=101

**Job Configuration**| **Description**
--- | ---
Base Git Hash | The Git Hash that you want to put your Gerrit patch on top of.
Gerrit URL | The patch you want to run the benchmark on. Patches in dependent repos (e.g. v8, skia) are supported. Pinpoint will also post updates on the Gerrit comment list.
Attempt Count | The number of iterations Pinpoint will run on both arms. Pinpoint will spread iterations evenly across all available devices. Pinpoint will also randomize which arm runs first and ensure that the number of iterations going first are the same for both arms.
Base Git Hash | The Git Hash of the control arm. Default is `HEAD`.
Exp Git Hash | The Git Hash of the experiment arm. Default is `HEAD`.
Base Patch | (optional) The patch you want the control arm to run the benchmark on. Patches in dependent repos (e.g. v8, skia) are supported. Pinpoint will also post updates on the Gerrit comment list.
Exp Patch | (optional) Same as Base Patch for the experiment arm.
Extra arguments on base commit | (optional) Extra arguments for the test. E.g. `--extra-chrome-categories="foo,bar"`<br><br>To see all arguments, run `tools/perf/run_benchmark run --help`
Extra arguments on experiment commit | (optional) Same as base commit for the experiment arm. Note that some arguments will apply to both arms.
Monorail Project | The repo the Git hashes are from. Default is `chromium`.
Bug ID | (optional) A bug ID. Pinpoint will post updates on the bug.
Batch ID | (optional) A batch ID used to track relevant jobs for the Chrome Health Initiative. We recommend leaving this blank.

## Interpreting the results

Expand Down

0 comments on commit f31a3e2

Please sign in to comment.