Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Polish up bazel karma saucelabs info tools/saucelabs/README.md #35667

Closed
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
29 changes: 22 additions & 7 deletions tools/saucelabs/README.md
Expand Up @@ -6,17 +6,28 @@ Setup your `SAUCE_USERNAME`, `SAUCE_ACCESS_KEY` & `SAUCE_TUNNEL_IDENTIFIER` envi

To run tests use:

```
``` bash
yarn bazel run //tools/saucelabs:sauce_service_setup
yarn bazel test //path/to:saucelabs_test_target_1 --config=saucelabs [--config=ivy]
yarn bazel test //path/to:saucelabs_test_target_2 --config=saucelabs [--config=ivy]
```

or if the tests are combined into a test suite:
`bazel query` is required gather up all karma saucelabs test labels so they can be run in one command as they are tagged `manual`.

Running all ViewEngine karma tests in Saucelabs:

``` bash
yarn bazel run //tools/saucelabs:sauce_service_setup
TESTS=$(./node_modules/.bin/bazel query --output label '(kind(karma_web_test, ...) intersect attr("tags", "saucelabs", ...)) except attr("tags", "ivy-only", ...) except attr("tags", "fixme-saucelabs-ve", ...)')
yarn bazel test --config=saucelabs ${TESTS}
```

Running all Ivy karma tests in Saucelabs:

``` bash
yarn bazel run //tools/saucelabs:sauce_service_setup
yarn bazel test //path/to:saucelabs_test_suite --config=saucelabs [--config=ivy]
TESTS=$(./node_modules/.bin/bazel query --output label '(kind(karma_web_test, ...) intersect attr("tags", "saucelabs", ...)) except attr("tags", "no-ivy-aot", ...) except attr("tags", "fixme-saucelabs-ivy", ...)')
yarn bazel test --config=saucelabs --config=ivy ${TESTS}
```

To see the test output while the tests are running as these are long tests, add the `--test_output=streamed` option. Note, this option will also prevent bazel from using the test cache and will force the test to run.
Expand All @@ -31,21 +42,25 @@ script which configures the saucelabs environment and starts Sauce Connect befor

For example,

```
``` python
karma_web_test(
name = "saucelabs_core_acceptance_tests",
timeout = "long",
karma = "//tools/saucelabs:karma-saucelabs",
tags = [
"exclusive",
"manual",
"no-remote-exec",
"saucelabs",
],
deps = [
"//packages/core/test/acceptance:acceptance_lib",
],
)
```

These saucelabs targets must be tagged `no-remote-exec` as they cannot be executed remotely since
they require a local Sauce Connect process. They should also be tagged `manual` so they are not
automatically tested with `//...`.
These saucelabs targets must have a few important tags:
* `no-remote-exec` as they cannot be executed remotely since they require a local Sauce Connect process
* `manual` so they are not automatically tested with `//...`
* `exclusive` as they must be run serially in order to not over-provision Saucelabs browsers
* `saucelabs` so that they can be easily gathered up for testing in a `bazel query`