Add automated browser testing #170

Merged
merged 2 commits into from Apr 30, 2017

Conversation

Projects
None yet
2 participants
@josephfrazier
Collaborator

josephfrazier commented Apr 11, 2017

This adds a test-saucelabs script to package.json that uses zuul to
run tests on browsers using SauceLabs.

Zuul is setup to test the browsers in the current browserslist defaults
list (shown at http://browserl.ist), constrained to what zuul
supports, according to:

./node_modules/.bin/zuul --list-available-browsers

(Opera 11 fails on SauceLabs, so skip it)


Travis CI is configured to run these tests if the SAUCE_USERNAME and
SAUCE_ACCESS_KEY environment variables are set. These credentials can
be obtained for free by signing up for the SauceLabs Open Source
program. Travis offers a way to set environment variables in the build
settings
, or they can be encrypted and added to the .travis.yml file.

Here's what a build looks like with this change:
https://travis-ci.org/josephfrazier/xregexp/builds/220784295


This also includes a slight refactor that moves beforeEach calls from
test helpers into spec files, because having them in the helpers doesn't
work with zuul. There is a test-browser script that can be used to run
zuul locally.

josephfrazier added some commits Feb 16, 2017

Add automated browser testing
This adds a `test-saucelabs` script to package.json that uses [zuul] to
run tests on browsers using [SauceLabs].

Zuul is setup to test the browsers in the current [browserslist] defaults
list (shown at <http://browserl.ist>), constrained to what zuul
supports, according to:

    ./node_modules/.bin/zuul --list-available-browsers

(Opera 11 fails on SauceLabs, so skip it)

---

Travis CI is configured to run these tests if the `SAUCE_USERNAME` and
`SAUCE_ACCESS_KEY` environment variables are set. These credentials can
be obtained for free by [signing up] for the SauceLabs Open Source
program. Travis offers a way to set environment variables in the [build
settings], or they can be [encrypted] and added to the .travis.yml file.

Here's what a build looks like with this change:
https://travis-ci.org/josephfrazier/xregexp/builds/220784295

---

This also includes a slight refactor that moves `beforeEach` calls from
test helpers into spec files, because having them in the helpers doesn't
work with zuul. There is a `test-browser` script that can be used to run
zuul locally.

[zuul]: https://github.com/defunctzombie/zuul
[SauceLabs]: https://saucelabs.com/open-source
[browserslist]: https://github.com/ai/browserslist
[signing up]: https://saucelabs.com/beta/signup/OSS/None
[build settings]: https://travis-ci.org/slevithan/xregexp/settings
[encrypted]: https://docs.travis-ci.com/user/environment-variables/#Defining-encrypted-variables-in-.travis.yml

@slevithan slevithan merged commit d83c5d2 into slevithan:master Apr 30, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@josephfrazier

This comment has been minimized.

Show comment
Hide comment
@josephfrazier

josephfrazier May 5, 2017

Collaborator

Thanks for merging! I just wanted to note that it looks like Saucelabs is having trouble running the tests for opera/ie/android, and is marking those platforms as failing (even though 0 tests failed). Here's some sample output from npm run test-saucelabs (see the failed lines):

- testing: chrome @ Windows 2008: 49 57
- testing: chrome @ Windows 2012 R2: 55
- testing: chrome @ Windows 2012: 56
- testing: firefox @ Windows 2012: 45 52
- testing: firefox @ Windows 2012 R2: 50
- testing: firefox @ Mac 10.11: 51
- testing: firefox @ Windows 10: 53
- testing: safari @ Mac 10.11: 9
- testing: safari @ Mac 10.12: 10
- testing: opera @ Windows 2003: 12
- testing: internet explorer @ Windows 2012: 10
- testing: internet explorer @ Windows 10: 11
- testing: microsoftedge @ Windows 10: 13 14
- testing: android @ Linux: 4.4 5.0 5.1 6.0
- testing: iphone @ Mac 10.11: 9.3 10.0 10.2
- testing: ipad @ Mac 10.11: 9.3 10.0 10.2
- queuing: <chrome 56 on Windows 2012>
- queuing: <chrome 55 on Windows 2012 R2>
- queuing: <chrome 57 on Windows 2008>
- queuing: <firefox 45 on Windows 2012>
- queuing: <chrome 49 on Windows 2008>
- starting: <chrome 57 on Windows 2008>
- starting: <chrome 49 on Windows 2008>
- starting: <chrome 55 on Windows 2012 R2>
- starting: <chrome 56 on Windows 2012>
- passed: <chrome 49 on Windows 2008>
- queuing: <firefox 50 on Windows 2012 R2>
- starting: <firefox 45 on Windows 2012>
- passed: <chrome 57 on Windows 2008>
- queuing: <firefox 51 on Mac 10.11>
- passed: <chrome 55 on Windows 2012 R2>
- queuing: <firefox 52 on Windows 2012>
- passed: <chrome 56 on Windows 2012>
- starting: <firefox 52 on Windows 2012>
- queuing: <firefox 53 on Windows 10>
- starting: <firefox 50 on Windows 2012 R2>
- passed: <firefox 45 on Windows 2012>
- queuing: <safari 9 on Mac 10.11>
- passed: <firefox 52 on Windows 2012>
- queuing: <safari 10 on Mac 10.12>
- passed: <firefox 50 on Windows 2012 R2>
- queuing: <opera 12 on Windows 2003>
- starting: <firefox 53 on Windows 10>
- starting: <firefox 51 on Mac 10.11>
- starting: <safari 10 on Mac 10.12>
- starting: <opera 12 on Windows 2003>
- starting: <safari 9 on Mac 10.11>
- passed: <firefox 53 on Windows 10>
- queuing: <internet explorer 10 on Windows 2012>
- starting: <internet explorer 10 on Windows 2012>
- passed: <safari 10 on Mac 10.12>
- passed: <firefox 51 on Mac 10.11>
- queuing: <microsoftedge 13 on Windows 10>
- failed: <opera 12 on Windows 2003> (0 failed, 0 passed)
- queuing: <microsoftedge 14 on Windows 10>
- passed: <safari 9 on Mac 10.11>
- queuing: <android 4.4 on Linux>
- queuing: <internet explorer 11 on Windows 10>
- failed: <internet explorer 10 on Windows 2012> (0 failed, 0 passed)
- queuing: <android 5.0 on Linux>
- starting: <microsoftedge 13 on Windows 10>
- starting: <internet explorer 11 on Windows 10>
- starting: <microsoftedge 14 on Windows 10>
- failed: <internet explorer 11 on Windows 10> (0 failed, 0 passed)
- queuing: <android 5.1 on Linux>
- passed: <microsoftedge 13 on Windows 10>
- queuing: <android 6.0 on Linux>
- passed: <microsoftedge 14 on Windows 10>
- queuing: <iphone 9.3 on Mac 10.11>
- starting: <android 4.4 on Linux>
- starting: <android 5.0 on Linux>
- failed: <android 4.4 on Linux> (0 failed, 0 passed)
- queuing: <iphone 10.0 on Mac 10.11>
- failed: <android 5.0 on Linux> (0 failed, 0 passed)
- queuing: <iphone 10.2 on Mac 10.11>
- starting: <android 5.1 on Linux>
- failed: <android 5.1 on Linux> (0 failed, 0 passed)
- queuing: <ipad 9.3 on Mac 10.11>
- starting: <android 6.0 on Linux>
- failed: <android 6.0 on Linux> (0 failed, 0 passed)
- queuing: <ipad 10.0 on Mac 10.11>
- starting: <iphone 9.3 on Mac 10.11>
- passed: <iphone 9.3 on Mac 10.11>
- queuing: <ipad 10.2 on Mac 10.11>
- starting: <ipad 9.3 on Mac 10.11>
- passed: <ipad 9.3 on Mac 10.11>
- starting: <iphone 10.2 on Mac 10.11>
- passed: <iphone 10.2 on Mac 10.11>
- starting: <ipad 10.0 on Mac 10.11>
- starting: <iphone 10.0 on Mac 10.11>
- passed: <ipad 10.0 on Mac 10.11>
- passed: <iphone 10.0 on Mac 10.11>
- starting: <ipad 10.2 on Mac 10.11>
- passed: <ipad 10.2 on Mac 10.11>
7 browser(s) failed
error Command failed with exit code 1.

Therefore, we should probably hold off on adding SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables to Travis CI until Saucelabs fixes these platforms (or we just remove them from .zuul.yml.

Collaborator

josephfrazier commented May 5, 2017

Thanks for merging! I just wanted to note that it looks like Saucelabs is having trouble running the tests for opera/ie/android, and is marking those platforms as failing (even though 0 tests failed). Here's some sample output from npm run test-saucelabs (see the failed lines):

- testing: chrome @ Windows 2008: 49 57
- testing: chrome @ Windows 2012 R2: 55
- testing: chrome @ Windows 2012: 56
- testing: firefox @ Windows 2012: 45 52
- testing: firefox @ Windows 2012 R2: 50
- testing: firefox @ Mac 10.11: 51
- testing: firefox @ Windows 10: 53
- testing: safari @ Mac 10.11: 9
- testing: safari @ Mac 10.12: 10
- testing: opera @ Windows 2003: 12
- testing: internet explorer @ Windows 2012: 10
- testing: internet explorer @ Windows 10: 11
- testing: microsoftedge @ Windows 10: 13 14
- testing: android @ Linux: 4.4 5.0 5.1 6.0
- testing: iphone @ Mac 10.11: 9.3 10.0 10.2
- testing: ipad @ Mac 10.11: 9.3 10.0 10.2
- queuing: <chrome 56 on Windows 2012>
- queuing: <chrome 55 on Windows 2012 R2>
- queuing: <chrome 57 on Windows 2008>
- queuing: <firefox 45 on Windows 2012>
- queuing: <chrome 49 on Windows 2008>
- starting: <chrome 57 on Windows 2008>
- starting: <chrome 49 on Windows 2008>
- starting: <chrome 55 on Windows 2012 R2>
- starting: <chrome 56 on Windows 2012>
- passed: <chrome 49 on Windows 2008>
- queuing: <firefox 50 on Windows 2012 R2>
- starting: <firefox 45 on Windows 2012>
- passed: <chrome 57 on Windows 2008>
- queuing: <firefox 51 on Mac 10.11>
- passed: <chrome 55 on Windows 2012 R2>
- queuing: <firefox 52 on Windows 2012>
- passed: <chrome 56 on Windows 2012>
- starting: <firefox 52 on Windows 2012>
- queuing: <firefox 53 on Windows 10>
- starting: <firefox 50 on Windows 2012 R2>
- passed: <firefox 45 on Windows 2012>
- queuing: <safari 9 on Mac 10.11>
- passed: <firefox 52 on Windows 2012>
- queuing: <safari 10 on Mac 10.12>
- passed: <firefox 50 on Windows 2012 R2>
- queuing: <opera 12 on Windows 2003>
- starting: <firefox 53 on Windows 10>
- starting: <firefox 51 on Mac 10.11>
- starting: <safari 10 on Mac 10.12>
- starting: <opera 12 on Windows 2003>
- starting: <safari 9 on Mac 10.11>
- passed: <firefox 53 on Windows 10>
- queuing: <internet explorer 10 on Windows 2012>
- starting: <internet explorer 10 on Windows 2012>
- passed: <safari 10 on Mac 10.12>
- passed: <firefox 51 on Mac 10.11>
- queuing: <microsoftedge 13 on Windows 10>
- failed: <opera 12 on Windows 2003> (0 failed, 0 passed)
- queuing: <microsoftedge 14 on Windows 10>
- passed: <safari 9 on Mac 10.11>
- queuing: <android 4.4 on Linux>
- queuing: <internet explorer 11 on Windows 10>
- failed: <internet explorer 10 on Windows 2012> (0 failed, 0 passed)
- queuing: <android 5.0 on Linux>
- starting: <microsoftedge 13 on Windows 10>
- starting: <internet explorer 11 on Windows 10>
- starting: <microsoftedge 14 on Windows 10>
- failed: <internet explorer 11 on Windows 10> (0 failed, 0 passed)
- queuing: <android 5.1 on Linux>
- passed: <microsoftedge 13 on Windows 10>
- queuing: <android 6.0 on Linux>
- passed: <microsoftedge 14 on Windows 10>
- queuing: <iphone 9.3 on Mac 10.11>
- starting: <android 4.4 on Linux>
- starting: <android 5.0 on Linux>
- failed: <android 4.4 on Linux> (0 failed, 0 passed)
- queuing: <iphone 10.0 on Mac 10.11>
- failed: <android 5.0 on Linux> (0 failed, 0 passed)
- queuing: <iphone 10.2 on Mac 10.11>
- starting: <android 5.1 on Linux>
- failed: <android 5.1 on Linux> (0 failed, 0 passed)
- queuing: <ipad 9.3 on Mac 10.11>
- starting: <android 6.0 on Linux>
- failed: <android 6.0 on Linux> (0 failed, 0 passed)
- queuing: <ipad 10.0 on Mac 10.11>
- starting: <iphone 9.3 on Mac 10.11>
- passed: <iphone 9.3 on Mac 10.11>
- queuing: <ipad 10.2 on Mac 10.11>
- starting: <ipad 9.3 on Mac 10.11>
- passed: <ipad 9.3 on Mac 10.11>
- starting: <iphone 10.2 on Mac 10.11>
- passed: <iphone 10.2 on Mac 10.11>
- starting: <ipad 10.0 on Mac 10.11>
- starting: <iphone 10.0 on Mac 10.11>
- passed: <ipad 10.0 on Mac 10.11>
- passed: <iphone 10.0 on Mac 10.11>
- starting: <ipad 10.2 on Mac 10.11>
- passed: <ipad 10.2 on Mac 10.11>
7 browser(s) failed
error Command failed with exit code 1.

Therefore, we should probably hold off on adding SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables to Travis CI until Saucelabs fixes these platforms (or we just remove them from .zuul.yml.

@josephfrazier josephfrazier deleted the josephfrazier:zuul-jasmine branch Jan 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment