Switch Meteor's headless browser tests from PhantomJS to Headless Chrome

[toinevk]

Per 254, this is an attempt to replace PhantomJS with Headless Chrome.

I evaluated a few different solutions notably:

• Puppeteer
• Chrome Launcher + Chrome Remote Interface
• Selenium

Ultimately, Puppeteer felt the best fit as it maintained by the Chrome team and it
downloads a recent version of Chromium that is guaranteed to work with the API. The only downside is that creates a fresh download (~170Mb Mac, ~282Mb Linux, ~280Mb Win) with every instance of the client. Chrome Launcher will instead find where Chrome is installed on your local file system and set up a debug instance. However, with this method the self-tester would assume that an instance of Chrome is installed that is compatible with the API.

There are some outstanding to dos, but before I proceed thought I would seek some feedback given Im not 100% familiar with Meteor homegrown test runner.

Outstanding To Dos:

• Add history entry
• dynamics-imports, modules and app-config tests all use dispatch-mocha-phantom-js.
• packages/test-in-console - Uses PhantomJS this needs to be removed.

[abernix]

This looks like a great direction. Thanks for taking the time to work on this!

The one thing we concretely rely on Phantom for, particularly as of Meteor 1.6.2 (soon to be 1.7!), is that it falls into the category of a "legacy browser" (that is to say, it lacks many "modern" browser features like async/await) which allows us to test the new legacy bundle features which 1.6.2 (now, 1.7) touts. (#9439) I still think it's probably worth moving forward on this this change though, and I like the direction and thoroughness you've got going here.

Do you think we could keep tools/tool-testing/clients/phantom/index.js (i.e. the PhantomClient class) in place, but make it so Puppeteer was the default? I think we can just keep the --phantom flag (here for the moment and simply add the --puppeteer flag as an addition. Then we can consider removing it later.

Overall, this PR would be best paired with the work I did in #9364, but try to get over the obstacle I encountered in #9364 (comment). If we got that working, then we could almost confidently remove Phantom, though it might still be worth keeping around for someone wanting to test "legacy browser" features without relying on an "online" service like BrowserStack.

Thanks again for diving into this! (Happy to help more.)

[toinevk]

Thanks @abernix, I have reverted the change and you can now pass --phantom to run test in the legacy browser. I have also updated the test-in-console runner to use puppeteer instead, note I had to bump the node version in Travis given puppeteer relies on at least version 6.4.0. I have pinned it at the current supported version for Meteor.

There is a single failure with a Blaze test on TravisCI, I dont have a-lot of familiarity here so I will need to unpack that to see if its valid...

[toinevk]

@abernix Is there a reliable way to run the tests in console package locally? I am unable to get it working reliable and the test in question is passing locally. Also given its a Blaze test its turning out tricky to debug! Any tips would be appreciated.

[abernix]

@toinevk Thanks for looking into this.

When I run ./packages/test-in-console/run.sh locally (the command ran here in .travis.yml), I encounter the same failure:

Error

C: tinytest - spacebars-tests - template_tests - javascript scheme urls : !!!!!!!!! FAIL !!!!!!!!!!!

Formatted Output

{
  "groupPath": ["tinytest", "spacebars-tests", "template_tests"],
  "test": "javascript scheme urls",
  "events": [{
    "sequence": 12317,
    "type": "fail",
    "details": {
      "type": "string_equal",
      "expected": "",
      "actual": "http://localhost:4096/"
    },
    "cookie": {
      "name": "spacebars-tests - template_tests - javascript scheme urls",
      "offset": 2,
      "groupPath": ["tinytest", "spacebars-tests", "template_tests"],
      "shortName": "javascript scheme urls"
    }
  }]
}

Which indicates that the error is happening when Puppeteer is running tests via the Chrome Inspector Protocol against its version of Chrome, which uses Chrome 67 (the so-called "Dev" Channel). (See chart of channel status here: https://googlechromelabs.github.io/current-versions/)

Then, instead of running the run-in-console version of the tests, I switched to using the normal "run-in-browser" technique by running:

./meteor test-packages -p 4096 ./packages/non-core/blaze/packages/spacebars-tests

I checked http://localhost:4096/ locally with what was my "Stable" channel (v65, before I restarted — oops, outstanding updates!) and the tests passed. But when I restarted Chrome to get the update, and advanced to v66, the tests started failing.

So basically, I think something may be changing/has changed in Chrome. If you're not experiencing the failure locally, I would try making sure that you've updated your Git submodules for Blaze locally, by running:

git submodule update --init --recursive

(You can check if you were on the wrong commit by going into packages/non-core/blaze and running git status, or by running git -C packages/non-core/blaze status from the root of the meteor checkout). The correct Git hash should be 8f819731df1fce254825c236176cfa5acb2df869.

[toinevk]

Thanks @abernix ! That information was super helpful (my version was also out of date...) - I managed to identify the issue and have submitted a PR at meteor/blaze#284 which goes into the detail. Once this is merged these tests should pass again.

[abernix]

Awesome, thanks for tracking that down @toinevk. That looks like exactly the culprit here.

[abernix]

@toinevk I've gone ahead and landed meteor/blaze#284 and updated the Blaze submodule within this PRs branch to include the update via my f2a4839 commit above.

It looks like we've solved the failing test, but another (likely legitimate?) failure seems to be occurring, as seen in the TravisCI test run:

(node:5310) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'newPage' of undefined
    at Timeout.runNextUrl [as _onTimeout] (/home/travis/build/meteor/meteor/packages/test-in-console/puppeteerRunner.js:4:30)
    at ontimeout (timers.js:482:11)
    at tryOnTimeout (timers.js:317:5)
    at Timer.listOnTimeout (timers.js:277:5)

[abernix]

I suspect that this runNextUrl function being called without a browser is the likely cause of the failure identified in #9814 (comment) and shown in this test run.

[toinevk]

Thanks @abernix, Ill put that down to late night coding... calling nextURL function was a hangover from the way PhantomJS is designed - its not needed for Puppeteer. Travis is passing now so hopefully CircleCI will too and it then should be good to go.

[abernix]

@toinevk I made a few changes above (and via meteor/circleci#1) in order to incorporate the additional packages into our existing CircleCI base image.

I thought we were pretty much ready to merge, but I re-enabled the --phantom flag for all CircleCI runs (with a0578e9), to ensure that we continue to test under both Phantom and Chrome per the above reasons, but it seems that's caused the failures above. Any ideas?

[toinevk]

@abernix It took a little while to isolate the issue, but in both instances it was the design of test which didn't account for multiple runs over the same instance of the test application. Given going forward the default locally will be one run and for CircleCI it will be 2, should we document this difference somewhere?

[benjamn]

Thanks so much for this substantial and valuable work @toinevk!

[abernix]

The final adjustments look great. I'd like to re-iterate the "thanks" that @benjamn has already stated above: Thank you very much, @toinevk!