test: saucelab targets for all karma tests

[gregmagolan]

PR Checklist

Please check if your PR fulfills the following requirements:

• The commit message follows our guidelines: https://github.com/angular/angular/blob/master/CONTRIBUTING.md#commit
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• angular.io application / infrastructure changes
• Other... Please describe:

What is the current behavior?

Issue Number: N/A

What is the new behavior?

Does this PR introduce a breaking change?

• Yes
• No

Other information

[josephperrott]

In talking with Igor, we realized that if we enable saucelabs for both Ivy and VE on all of our PRs, we run into concurrency issues causing resource contention. We likely need to run a subset of these tests for all PRs and on master and do the full run on a cron each day.

[gregmagolan]

SGTM. I'll list the all the tests here and we can pick the ones (if any) we want to run in PR. The Ivy & VE jobs can be run sequentially to reduce resource contention (tho you'll still get that for when multiple PRs are building).

[gregmagolan]

Here all of the tests currently running and how long they take:

ViewEngine (25 minutes)

INFO: Elapsed time: 1380.399s, Critical Path: 135.52s
//packages/animations/browser/test:saucelabs_test_web                    PASSED in 53.5s
//packages/animations/test:saucelabs_test_web                            PASSED in 45.4s
//packages/common/http/test:saucelabs_test_web                           PASSED in 42.0s
//packages/common/http/testing/test:saucelabs_test_web                   PASSED in 46.0s
//packages/common/test:saucelabs_test_web                                PASSED in 51.6s
//packages/compiler/test:saucelabs_test_web                              PASSED in 53.2s
//packages/compiler/test/css_parser:saucelabs_css_parser_web             PASSED in 45.9s
//packages/compiler/test/expression_parser:saucelabs_expression_parser_web PASSED in 44.0s
//packages/compiler/test/ml_parser:saucelabs_ml_parser_web               PASSED in 44.3s
//packages/compiler/test/selector:saucelabs_selector_web                 PASSED in 44.9s
//packages/core/test:saucelabs_test_web                                  PASSED in 97.9s
//packages/core/test/acceptance:saucelabs_acceptance_web                 PASSED in 77.9s
//packages/core/test/render3:saucelabs_render3_web                       PASSED in 45.2s
//packages/core/test/view:saucelabs_view_web                             PASSED in 54.1s
//packages/forms/test:saucelabs_test_web                                 PASSED in 55.1s
//packages/http/test:saucelabs_test_web                                  PASSED in 53.7s
//packages/platform-browser-dynamic/test:saucelabs_test_web              PASSED in 48.7s
//packages/platform-browser/animations/test:saucelabs_test_web           PASSED in 67.1s
//packages/platform-browser/test:saucelabs_test_web                      PASSED in 48.4s
//packages/platform-webworker/test:saucelabs_test_web                    PASSED in 47.4s
//packages/router/upgrade/test:saucelabs_test_web                        PASSED in 44.3s
//packages/upgrade/src/common/test:saucelabs_test                        PASSED in 49.9s
//packages/upgrade/src/dynamic/test:saucelabs_test                       PASSED in 61.6s
//packages/upgrade/static/test:saucelabs_test                            PASSED in 64.7s
//packages/upgrade/static/testing/test:saucelabs_test                    PASSED in 49.6s

Ivy (19 minutes)

INFO: Elapsed time: 1036.857s, Critical Path: 121.68s
//packages/animations/browser/test:saucelabs_test_web                    PASSED in 45.2s
//packages/animations/test:saucelabs_test_web                            PASSED in 44.6s
//packages/common/http/test:saucelabs_test_web                           PASSED in 47.4s
//packages/common/http/testing/test:saucelabs_test_web                   PASSED in 45.5s
//packages/common/test:saucelabs_test_web                                PASSED in 50.8s
//packages/compiler/test/css_parser:saucelabs_css_parser_web             PASSED in 47.1s
//packages/compiler/test/expression_parser:saucelabs_expression_parser_web PASSED in 66.2s
//packages/compiler/test/ml_parser:saucelabs_ml_parser_web               PASSED in 45.1s
//packages/compiler/test/selector:saucelabs_selector_web                 PASSED in 44.5s
//packages/core/test/acceptance:saucelabs_acceptance_web                 PASSED in 63.6s
//packages/core/test/render3:saucelabs_render3_web                       PASSED in 43.3s
//packages/http/test:saucelabs_test_web                                  PASSED in 45.3s
//packages/platform-browser-dynamic/test:saucelabs_test_web              PASSED in 45.9s
//packages/platform-browser/test:saucelabs_test_web                      PASSED in 48.6s
//packages/platform-webworker/test:saucelabs_test_web                    PASSED in 46.9s
//packages/router/upgrade/test:saucelabs_test_web                        PASSED in 42.4s
//packages/upgrade/src/common/test:saucelabs_test                        PASSED in 44.6s
//packages/upgrade/src/dynamic/test:saucelabs_test                       PASSED in 51.8s
//packages/upgrade/static/test:saucelabs_test                            PASSED in 57.6s
//packages/upgrade/static/testing/test:saucelabs_test                    PASSED in 46.5s

NB: See the PR diff for the few that are failing on IE 10 & 11

These can all get moved to a master-only cron job. Question to answer is which ones (if any) do we want to run on all PRs.

[josephperrott]

For now, lets just run the same subset of tests we were running before on PRs and on master, and then on a cron we can set up to run them all.

We should have the cron notify the dev-infra failures channel if it fails.

[gregmagolan]

Sounds good. Currently legacy saucelabs test runs around 6000 specs in a single karma config which takes around 7m (https://app.circleci.com/jobs/github/angular/angular/628633). I suspect that running the same specs across multiple karma configs will take longer with the extra browser provisioning & teardown overhead. I'll account for which ones are being run right now and compare to what is run in the current 25m ViewEngine bazel saucelabs job that runs 25 different karma_web_test targets.

@devversion @josephperrott I wonder it would be possible to tell the karma-sauce-launcher (https://github.com/karma-runner/karma-sauce-launcher) to keep browsers it provisioned open for some amount of time so that they could be re-used in the tunnel by the next karma-sauce-launcher?

[gregmagolan]

Moved the two bazel saucelabs jobs (Ivy & ViewEngine) to master-only and kept the legacy_saucelabs job for now until we can fix the browsering provisioning with Bazel. CI tested for the master-only jobs here: #35643

The question is if we should run the two saucelabs jobs on each master commit or on a cron job?

[josephperrott]

LGTM

[gregmagolan]

Ready to go. #35643 all green

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}