Skip to content

Commit

Permalink
Formatted and removed lints from devicelab README.md (#117239)
Browse files Browse the repository at this point in the history
  • Loading branch information
gaaclarke committed Dec 16, 2022
1 parent ebeb491 commit 93c581a
Showing 1 changed file with 37 additions and 28 deletions.
65 changes: 37 additions & 28 deletions dev/devicelab/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,29 +7,36 @@ the tests are referred to as "tasks" in the API, but since we primarily use it
for testing, this document refers to them as "tests".

Current statuses for the devicelab are available at
https://flutter-dashboard.appspot.com/#/build. See [dashboard user guide](https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md)
<https://flutter-dashboard.appspot.com/#/build>. See [dashboard user
guide](https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md)
for information on using the dashboards.

## Table of Contents

* [How the DeviceLab runs tests](#how-the-devicelab-runs-tests)
* [Running tests locally](#running-tests-locally)
* [Writing tests](#writing-tests)
* [Adding tests to continuous integration](#adding-tests-to-continuous-integration)
* [Adding tests to continuous
integration](#adding-tests-to-continuous-integration)
* [Adding tests to presubmit](#adding-tests-to-presubmit)


## How the DeviceLab runs tests

DeviceLab tests are run against physical devices in Flutter's lab (the "DeviceLab").
DeviceLab tests are run against physical devices in Flutter's lab (the
"DeviceLab").

Tasks specify the type of device they are to run on (`linux_android`, `mac_ios`, `mac_android`, `windows_android`, etc).
When a device in the lab is free, it will pickup tasks that need to be completed.
Tasks specify the type of device they are to run on (`linux_android`, `mac_ios`,
`mac_android`, `windows_android`, etc). When a device in the lab is free, it
will pickup tasks that need to be completed.

1. If the task succeeds, the test runner reports the success and uploads its performance metrics to Flutter's infrastructure. Not
all tasks record performance metrics.
2. If task fails, an auto rerun happens. Whenever the last run succeeds, the task will be reported as a success. For this case,
a flake will be flagged and populated to the test result.
3. If the task fails in all reruns, the test runner reports the failure to Flutter's infrastructure and no performance metrics are collected
1. If the task succeeds, the test runner reports the success and uploads its
performance metrics to Flutter's infrastructure. Not all tasks record
performance metrics.
2. If task fails, an auto rerun happens. Whenever the last run succeeds, the
task will be reported as a success. For this case, a flake will be flagged and
populated to the test result.
3. If the task fails in all reruns, the test runner reports the failure to
Flutter's infrastructure and no performance metrics are collected

## Running tests locally

Expand Down Expand Up @@ -63,10 +70,11 @@ To run a test, use option `-t` (`--task`):

Where `NAME_OR_PATH_OF_TEST` can be either of:

- the _name_ of a task, which is a file's basename in `bin/tasks`. Example: `complex_layout__start_up`.
- the path to a Dart _file_ corresponding to a task, which resides in `bin/tasks`.
Tip: most shells support path auto-completion using the Tab key. Example:
`bin/tasks/complex_layout__start_up.dart`.
* the _name_ of a task, which is a file's basename in `bin/tasks`. Example:
`complex_layout__start_up`.
* the path to a Dart _file_ corresponding to a task, which resides in
`bin/tasks`. Tip: most shells support path auto-completion using the Tab key.
Example: `bin/tasks/complex_layout__start_up.dart`.

To run multiple tests, repeat option `-t` (`--task`) multiple times:

Expand Down Expand Up @@ -107,19 +115,19 @@ Example:

The `--ab=10` tells the runner to run an A/B test 10 times.

`--local-engine=host_debug_unopt` tells the A/B test to use the `host_debug_unopt`
engine build. `--local-engine` is required for A/B test.
`--local-engine=host_debug_unopt` tells the A/B test to use the
`host_debug_unopt` engine build. `--local-engine` is required for A/B test.

`--ab-result-file=filename` can be used to provide an alternate location to output
the JSON results file (defaults to `ABresults#.json`). A single `#` character can be
used to indicate where to insert a serial number if a file with that name already
exists, otherwise, the file will be overwritten.
`--ab-result-file=filename` can be used to provide an alternate location to
output the JSON results file (defaults to `ABresults#.json`). A single `#`
character can be used to indicate where to insert a serial number if a file with
that name already exists, otherwise, the file will be overwritten.

A/B can run exactly one task. Multiple tasks are not supported.

Example output:

```
```text
Score Average A (noise) Average B (noise) Speed-up
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 2900.20 (8.44%) 2426.70 (8.94%) 1.20x
bench_card_infinite_scroll.canvaskit.totalUiFrame.average 4964.00 (6.29%) 4098.00 (8.03%) 1.21x
Expand All @@ -142,13 +150,14 @@ Summarize tool example:
ABresults.json ABresults1.json ABresults2.json ...
```

`--[no-]tsv-table` tells the tool to print the summary in a table with tabs for easy spreadsheet
entry. (defaults to on)
`--[no-]tsv-table` tells the tool to print the summary in a table with tabs for
easy spreadsheet entry. (defaults to on)

`--[no-]raw-summary` tells the tool to print all per-run data collected by the A/B test formatted
with tabs for easy spreadsheet entry. (defaults to on)
`--[no-]raw-summary` tells the tool to print all per-run data collected by the
A/B test formatted with tabs for easy spreadsheet entry. (defaults to on)

Multiple trailing filenames can be specified and each such results file will be processed in turn.
Multiple trailing filenames can be specified and each such results file will be
processed in turn.

## Reproducing broken builds locally

Expand Down Expand Up @@ -208,7 +217,7 @@ _TASK_- the name of your test that also matches the name of the

1. Add target to
[.ci.yaml](https://github.com/flutter/flutter/blob/master/.ci.yaml)
- Mirror an existing one that has the recipe `devicelab_drone`
* Mirror an existing one that has the recipe `devicelab_drone`

If your test needs to run on multiple operating systems, create a separate
target for each operating system.
Expand Down

0 comments on commit 93c581a

Please sign in to comment.