Support modern browsers, not just PhantomJS

[bergie]

PhantomJS 2, uses WebKit 538.1, which was released in November 2013.
Significant improvements have been done in browsers in these 2+ years. Both in terms of performance, new HTML apis (not so relevant for us) and new language support (ES6/ES2015 etc).
Because Qt Project deprecated the QtWebkit they use in favor of Blink-based QtWebEngine, a more modern version would only come with (yet unplanned) PhantomJS 3 which would rewrite this...

The challenge is to have a headless setup suitable for running in production.

Alternatives

• Chromium (Blink). Communicate through WebDriver/Selenium, or run-headless-chrome
• Firefox (Gecko). Communicate through WebDriver/Selenium, or SlimerJS
• Servo. Communicate/setup through??
  Fast but highly experiemental webengine, part of Mozilla's Rust project

Headless approaches

• Get xfvb-run (virtual X11) to work on Heroku. Possible but not easy
• Use some virtual/headless Wayland compositor instead of X11. Weston headless. But neither Firefox nor Chromium has good mainline support yet.
• Use the native headless support in Servo
• Maybe try Ozone headless support in Chromium Content Shell. Looks not very mature
• Run Docker images with virtual X11. On Amazon AWS ECS or similar, instead of Heroku.
• Build our own headless browser shell, for instance based on QtWebEngine or Chromium Embedding framework
• Native headless backend for Firefox or for Chromium. Large undertaking

Related to #6

[jonnor]

Chromium headless looks to have been worked on actively the last couple of months. Instructions on how to build here: https://chromium.googlesource.com/chromium/src/+/master/headless/README.md

It will require a custom build Chromium for foreseeable future, since graphics backend is compile time choice and Linux builds use X11 or Wayland probably.

[jonnor]

Looks like QtWebkit has been resurrected, with ES2015 etc coming to PhantomJS 2.5:
https://groups.google.com/forum/m/#!topic/phantomjs/AefOuwkgBh0

[bergie]

Chrome headless will be shipping in 59 for all platforms:

https://www.chromestatus.com/features/5678767817097216

[jonnor]

And the PhantomJS maintainer stepped down, https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE

[jonnor]

Headless API for Chrome, by official Chrome team and using a standard binary build of Chrome: https://github.com/GoogleChrome/puppeteer

[jonnor]

Some work is still going on in PhantomJS, moving towards a 2.5 release: ariya/phantomjs#14458