Run tests in all supported browsers on BrowserStack

[ovidiu1]

Note: if i'm running using Chrome ( bs_win10_chrome_latest) all the tests are running.

I try to run all the tests on IE11 (bs_win10_ie_11), and i got something like this:

λ npm run test:bs

> @lion/root@ test:bs D:\dev\lab\lion
> karma start karma.bs.config.js --coverage


START:
18 06 2019 21:28:51.979:WARN [filelist]: Pattern "D:/dev/lab/lion/__snapshots__/**/*.md" does not match any file.
18 06 2019 21:28:53.547:INFO [karma-server]: Karma v4.1.0 server started at http://0.0.0.0:9876/
18 06 2019 21:28:53.548:INFO [launcher]: Launching browsers bs_win10_ie_11 with concurrency unlimited
18 06 2019 21:28:53.607:INFO [launcher]: Starting browser IE 11.0 (Windows 10) on BrowserStack
18 06 2019 21:29:00.208:INFO [launcher.browserstack]: IE 11.0 (Windows 10) session at https://automate.browserstack.com/builds/a48343450ca0f66a4f1ac1da96d4536c314fe859/sessions/69d851431e3a82dc9cc70a483c21ff536a188c53
18 06 2019 21:29:05.135:INFO [IE 11.0.0 (Windows 10.0.0)]: Connected on socket RDJX-pPfYMpMcPdXAAAA with id 79747197

Finished in 0.003 secs / 0 secs @ 21:29:07 GMT+0300 (Eastern European Summer Time)

SUMMARY:
√ 0 tests completed

=============================== Coverage summary ===============================
Statements   : Unknown% ( 0/0 )
Branches     : Unknown% ( 0/0 )
Functions    : Unknown% ( 0/0 )
Lines        : Unknown% ( 0/0 )
================================================================================
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @lion/root@ test:bs: `karma start karma.bs.config.js --coverage`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @lion/root@ test:bs script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

I think is a problem with browserstack, also i found this issue.

[bashmish]

There are 2 issues here.

1. You need to use a legacy config, so you need to pass --legacy to karma like we do in npm run test:legacy. We might need to add another NPM script for that like npm run test:legacy:bs and use that script in the CI in addition to the current one.
2. The list of browsers needs to respect the --legacy argument as well.

      browsers: [
        // TODO: change this depending on whether the `--legacy` argument is provided
        // 'bs_win10_chrome_latest',
        // 'bs_win10_firefox_latest',
        // 'bs_win10_edge_latest',
        // 'bs_osxmojave_safari_latest',
        // 'bs_win10_ie_11',
      ],

And the this is done, you'll run into some timeouts and flaky tests we have in some browsers including IE11. It is not many and we are planning on making them work consistently in the future. You can also help with that if you want to.

[bashmish]

Some notes from the discussion here #120

:legacy is for testing transpiled code everywhere, not only in old browsers. So the final logic should be like this:

1. modern config runs in browsers that support ES modules (modern Chrome, Firefox, Safari, Edge)
2. legacy config runs in all browsers (the ones above + IE11)

The reason it needs to be like this is because often the code is transpiled and run in production on every browser, while some people might have differential builds and serve different code based on browser agent sniffing. So basically transpiled code needs to work everywhere. And modern code should work where possible, because users might load it for modern browsers, e.g. to improve performance.

[owen-m1]

I sent a support inquiry to Browserstack about this after I was linked here from the issue mentioned in the OP. I will let you know their response if I get one.

[erikkroes]

@owen-m1 Any update on this?

[LarsDenBakker]

Compatibility mode in @open-wc/testing-karma is now automatic, the same command can run all browsers with efficiency.

[owen-m1]

@erikkroes In the end I switched to SauceLabs

[ovidiu1]

We can close this.