Browser testing

[ctavan]

Follow-up for #341 but still WIP!

• Currently the examples install uuid from npm (in the current state it's even a github repo where I uploaded the npm package content just for testing, but I would change that to an npm package version). How would we run the test before releasing a new version to npm? -> Now using local dist build.
• Define a browser matrix and run the test for the browser we want to support.
• Maybe extract the setup and teardown code. -> for the time being I don't see much room for improvement.
• Maybe only run this on master/next but not for PRs from forks. -> Not for now.

[ctavan]

@broofa I would be interested in some preliminary feedback on this one (it's not ready for approval yet).

Here's what it currently does:

• define a separate github workflow: https://github.com/kelektiv/node-uuid/pull/348/checks?check_run_id=333927622
• build all examples/
• set up an http-server to serve this directory
• set up browserstack-local make the local http server available to browserstack.
• load the html files from the examples through browserstack using selenium webdriver and run assertions on the rendered output: https://github.com/kelektiv/node-uuid/pull/348/files#diff-fc9324a2435c78be81668dd578da863c

That way we should get end-to-end assertions that uuid works as expected when used with webpack/rollup/etc. in the browser.

Open questions:

• Currently the examples install uuid from npm (in the current state it's even a github repo where I uploaded the npm package content just for testing, but I would change that to an npm package version). How would we run the test before releasing a new version to npm?
• What do you think about the approach of rendering uuid() output into html and the running assertions on it through selenium?

What I don't really struggle with, but still have to do:

• Define a browser matrix and run the test for the browser we want to support.
• Maybe extract the setup and teardown code.
• Maybe only run this on master/next but not for PRs from forks.

[ctavan]

@broofa I think I've done quite some progress on this one now and I believe it's ready for a more serious review!

What we get here is:

• a basic test of the module API when used with webpack and rollup
• all tests run in a set of browser that we intend to support (minimum and latest versions)
• tests are run in Browserstack, with all browsers running in parallel

When defining the browser support matrix I somewhat arbitrarily chose the minimum browser version that supports both let/const and crypto.getRandomValues().

Since the tests do take quite a while (~7-15 mins) to run the only thing I'm still wondering is when to execute these tests. I have also experienced that these test, when run from Github actions, sometimes time out which I haven't experienced yet when running them from my local machine.

So all in all running them on every single pull requests seems wasteful and potentially a bit brittle, after all we mostly want to ensure that we don't introduce regressions, so maybe running them before each release would be enough? If so, how would we achieve that? What do you think?

[broofa]

Hey, this looks really good! Thanks for tackling this.

Add the following to .gitignore? Looks like npm run test:browser generates these.

# Browserstack
browserstack.err
local.log

So all in all running them on every single pull requests seems wasteful and potentially a bit brittle

Regarding "wasteful", do we foresee enough PR churn for that to really matter? 'Seems like the benefit of having CI tests pass as a prerequisite for merging a PR would outweigh the cost there... but, that depends on the cost. FWIW, I'm on one BS's opensource accounts so it doesn't cost me anything. That expires in April 2020, though. (I assume I can ask them to renew it, but have yet to go through that process with them.)

We could potentially see about getting an organization account of some sort I suppose...?

I'm not familiar enough with Github actions to know if there's a way to tear down a workflow if/when the PR is updated, so any BS tests in progress get cancelled and restarted? Or maybe that happens automatically? (Seems like an obvious thing to do.)

Regarding "brittle", the timeout issue is a little concerning but I think it would be better for all concerned if we figured out how to make CI testing work in the cloud rather than relying on the vagaries of individual contributor environments.

tl;dr: Triggering CI tests on each commit is nice.

[ctavan]

Thanks for the feedback!

So let's start out by running the Browserstack-Tests on all PRs and merges just like the regular tests and let's see if we can either fix brittleness or if it doesn't even turn out to be problematic. They run on my OpenSource account as well (still valid until April 2021 I think) so they won't incur actual cost.

Re gitignore: done, thanks for finding that one!

[ctavan]

OK, so while I was hit by a brittle browser test run (where only tests in Firefox 70 timed out for whatever reason), I figured out that it is actually faster to run the different browsers as a github actions matrix.

I was hoping that this would also allow us to re-trigger specific failed browser individually but that doesn't seem to be the case.