Use SauceLabs for e2e browser integrations tests

[zouhir]

Cypress.io has a nice API for writing automated browser test but it does not support Safari and support for FireFox is in Beta, that makes Saucelabs the best option for automated browser testing at this moment.

SauceLabs testing strategies:

1: The Web-animations-js polyfill strategy**

That polyfill have a mix of: JavaScript unit tests and e2e integration tests, all run in CI via Saucelabs which we are planning to use for ScrollTimeline.

Unit tests written in Mocha and running against real browser APIs (not JSDOM) via Karma test runner.

I believe the e2e integration test setup (for WPT, Blink test files) is counter-intuitive and hard to maintain, for those reasons:

• Karma excels at running JS tests against DOM or web pages as a context, and not running tests in HTML files.
• Web-animations-js polyfills has a long chain of configurations: a framework-specific task runner modules that pre-processes target files and has a modified testharnessreporter (forked from WPT) as well as custom Karma test harness runners you can see here. Being as close as possible to WPT files is beneficial and the more pre-processing and customization, the harder it will be for us to maintain.

2: Philip Walton's Blog Post on using Saucelabs Rest API read

The solution outline in Philip Walton's (Engineer @ Google) Blog is more straight-forward than Web-animations-js polyfill solution, he uses Mocha (the client side library) to run tests inside the HTML pages, and post those pages to sauce labs using their REST API, all needed there is a custom "failLogger" when the tests are finished, which is 1 JS function that acts as an adapter between Mocha results and Saucelabs.

One note is that the Saucelabs rest API does not support localhost:xxxx addresses, the developer in that post used ngrok and mentioned using Sauce Connect VPN tunnel if it's possible

3: Using Webdriver with Saucelabs read

I found connect WPT from lcoalhost:xxxx to Saucelabs via a sauce connect tunnel and using selenium web driver to test against a matrix of browsers is the simplest & most effective way to run WPT harness tests, and even do more if needed (screenshots, ref, etc).

Some of the pros:

• not much pre-processing needed: we only need to inject the polyfill script below the WPT/**/*.html test </title>
• all tests are ran via WPT test runner, no customization to the runner or reporter.

4: WPT test runner --sauce flag

Monday, April 20, Stephen McGruer from the Chromium project highlighted on the slack channel that we can use --sauce-* flags, they are documented here I believe if this works after we inject the HTML, will even be an improvements to saucelabs' Web Driver implementation

---

This is a child task of: #3