Research JavaScript Testing Packages

[rfultz]

Let's look into options and choose a new JavaScript testing package, to replace PhantomJS at a minimum.

What are the pain points for our current setup?

• Our current setup uses PhantomJS to test pages in the browser, which
  • doesn't support ES6
  • has been abandoned by its developers
  • is blocking us from upgrading our entire chain of software (Karma > Node > …)
• Using older software versions
  • decreases the security of our site and tools
  • takes more time because we can't use more modern approaches
  • limits our growth as a team and as professionals

What features do we require?

• Unit tests (functions and classes for expected output (and error handling))
• Integration tests (processes and components to behave as expected (and side effects))
• UI tests (tests across browsers and devices)
• Coverage tests (how much of our code is being used/called)
• Testing in at least Chrome
• Linting warnings & errors
• npm vulnerability warnings & errors

What would be nice to have?

• Testing more than only Chrome
• Testing for mobile platforms
• Screenshots of various pages' layouts
• Keeping up with updates would help us keep up with security concerns

Our current setup:

Karma
    Loads plugins
        karma-browserify
        karma-chai-sinon
        karma-chrome-launcher
        karma-coverage
        karma-coverage-istanbul-reporter
        karma-mocha
        karma-phantomjs-launcher
        karma-phantomjs-shim
        karma-webpack
    Launches
        Chrome
        [currently no others]
    Frameworks
        browserify
        phantomjs-shim
        mocha
        chai-sinon
    Preprocessors
        browserify
        webpack
            loaders
                babel-loader
                istanbul-instrumenter-loader
    Reporters
        progress (default, included with Karma)
        coverage-istanbul
        coverage (if --debug)

Do we have any compatibility issues?

• Our current setup requires versions of Karma, Node, etc that are a few versions from latest

Options

Browserless (Git)

Support:
Capabilities:
Limitations: It's a subscription service
Notes: Built on top of Puppeteer

Cypress (Git)

Support: 86 Git contributors, several commits a day
Capabilities: Testing suite that seems to incorporate Chai, Mocha. Can do screenshots, too
Limitations: More of a test runner like Karma than a direct replacement for PhantomJS.
Notes:

Jest (Git)####

Support: Facebook
Capabilities: Unit and integration tests; would take over a lot from Karma
Limitations: More of a general testing platform than a specific headless browser
Notes: Seems like it wants to do more of Webpack's job that we'd currently give it

Jsdom

Support: 229 Git contributors, release code every 2-5 weeks
Capabilities: Headless DOM
Limitations: Doesn't test in actual browsers
Notes: Interesting but seems like it's more for React components than functions and UI tests

NW.js (Git)

Support: 96 Git contributors, commit code every couple days
Capabilities: Lets us call Node modules from within the browser
Limitations:
Notes: Doesn't seem like what we need

Puppeteer (Git)

Support: Puppeteer has been acquired by Google so we should have stronger and stronger support for a while. Puppeteer has become the basis for several other testing packages. 193 Git contributors. Looks like there are updates every day
Capabilities: Typical headless browser, but can generate screenshots and PDFs, timeline trace to help diagnose performance issues
Limitations: Only tests in Chrome
Notes: Chrome is the most-used browser by a long shot. Soon, IE/Edge will be powered by Chromium. We currently only test in Chrome (though we should expand that) so we wouldn't be losing anything. This is the leading candidate as of concluding research, before team discussion

SlimerJS (Git)

Support: 62 Git contributors. Latest commit was May 2018
Capabilities: Headless browser that tried to fully mimic PhantomJS but with Gecko and SpiderMonkey—and Slimer fully supports ES6
Limitations: Uses Gecko, the Firefox engine
Notes: Seems promising as part of a broader testing platform, though the lack of updates is a concern

WebDriver / Selenium

Support: Controls any web browser
Capabilities: Standard, if limited testing automation
Limitations: Features are limited to those that apply to every browser. Updates are slow to roll out
Notes: Good that it can be used with any browser but not great that it's slow to update

xvfb

Is a virtual framebuffer—essentially a browser window that never draws to the screen
Support:
Capabilities:
Limitations:
Notes: Often mentioned in discussions of automated testing but doesn't seem like what we need

Zombie.js (Git)

Support: 128 Git contributors, seems like there's an update every couple months but have been more and way less frequent
Capabilities: assert, browser controls, cookies, tabs
Limitations: Doesn't seem to support typical browsers. Can't save screenshots
Notes:

Other options and considerations

Now that we're using a new Node, should we look into alternatives for Karma (like using Node to sequence tests itself)?

[rfultz]

Puppeteer is interesting

[rfultz]

Possibly related to this: #2653

[rfultz]

Tagging @johnnyporkchops @lbeaufort and @patphongs for review, comments, etc

[rfultz]

@johnnyporkchops, @patphongs, and I met to discuss these findings. We decided to go with Puppeteer for now to get tests in and working, and keep an eye on other options to expand testing for other browsers. Closing the ticket now.