Make ScalaWebTest compatible with all implementations of Selenium WebDriver

[DaniRey]

If we do not depend on HtmlUnits HTML parsing, we can use other Drivers as well and even extract the gauge logic completely. Ideally, we can compile this for Scala.js and use it in other tools like TestCafé. This will not be possible, as we have to parse HTML (see next comment). Most probably a rewrite for JS is the better option, if we want to use Gauges in TestCafé.

[DaniRey]

This is a challenging task because:

• org.openqa.selenium.WebElement is very limited. There is no way to query its parent, siblings or children (no DOM traversal)
• WebDriver does not provide access to Response Headers and Response Code

Because of this, a few intermediate steps are needed.

• not being able to use com.gargoylesoftware.htmlunit.html.HtmlElement means we have to parse the HTML in ScalaWebTest
• not many options exist to parse HTML. So far JSoup seems to be the best option
• to get a hold of Response Headers and Response Code we have to send all requests through a proxy, which is under our control. We can then ask the proxy to get this information. (inspiration https://github.com/wkeeling/selenium-wire)

[DaniRey]

I think the final solution should provide three different kinds of Browsers

• HtmlUnitExtended (default)
• Selenium
• ProxiedSelenium

All three should implement a common interface, but similar to JDBC drivers, some methods might not be implemented, pointing to alternatives which implement said method.

HtmlUnitExtended: This webdriver is headless. It uses HtmlUnit and doesn't hide that fact.

No system prerequisites	Response Codes	Response Headers	Screenshots	JavaScript
Yes	Yes	Yes	No	Limited (RhinoJs)

Selenium: This might be Chrome, Firefox,... controlled via SeleniumWebDriver API

No system prerequisites	Response Codes	Response Headers	Screenshots	JavaScript
Browser and WebDriver installed	No	No	Yes	Yes

Selenium Proxied: Again Chrome, Firefox... controlled via SeleniumWebDriver API and using a proxy, which is under our control. This allows ScalaWebTest to read the response code and response headers. This information is not exposed by the Selenium WebDriver API and never will (see SeleniumHQ/selenium-google-code-issue-archive#141 (comment)).

No system prerequisites	Response Codes	Response Headers	Screenshots	JavaScript
Browser and WebDriver installed and Proxy running	Yes	Yes	Yes	Yes

[DaniRey]

Following the reasoning from ashley.leyba in SeleniumHQ/selenium-google-code-issue-archive#141 (comment)

If we attempt to make WebDriver the silver bullet of web testing, then it will suffer in overall quality.

1. driver.get(url) blocks until the browser has loaded the page. In the case of a login redirect, what status/headers would you want returned? By the semantics of the API, we would return the response for the final loaded page. You would end up with a 200 instead of the 302 you're looking for.

3. If you want to test HTTP request/response headers, then write an appropriately sized test and test those directly.

We should probably not add Selenium Proxied to ScalaWebTest. I can currently not think of any reasonable use case, where one would require to use Chrome/Firefox/... to test a response code or response header. For those tests, HtmlUnitExtended should be enough. Therefore I would concentrate our efforts on HtmlUnitExtended and Selenium. The ProxiedSelenium variant might follow as a module, but I doubt we will ever need it.

[DaniRey]

We should try to use or integrate with https://github.com/bonigarcia/webdrivermanager

[DaniRey]

ScalaWebTest is now compatible with any Selenium WebDriver. It still uses HtmlUnit as it's default. Also some features, like verifying the responseCode or disabling JavaScript, are only available with HtmlUnit (because the Selenium WebDriver API does not allow to control or access this information).

At the moment we provide an implementation to use Chrome. Additional implementations are planned. At least Firefox, probably Safari as well. Our plan is to support all browsers, which we can test with Travis. We do not want to add implementations, which we cannot test as part of our CI build. As ScalaWebTest now works against the Selenium WebDriver API, users are free to plug any Selenium WebDriver, when using ScalaWebTest.

[DaniRey]

Firefox implementation was tried with https://github.com/unic/ScalaWebTest/blob/feature/add_firefox

The implementation itself works well, but the build on TravisCI is very unstable with Firefox added.
The log is cluttered with

(firefox:6123): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Unable to autolaunch a dbus-daemon without a $DISPLAY for X11

And Firefox sometimes fails with

[info] org.scalawebtest.integration.browser.SeleniumFirefoxSpec *** ABORTED ***
[info]   org.openqa.selenium.JavascriptException: TypeError: document.documentElement is null
[info] Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
[info] System info: host: 'travis-job-ce3fd6ff-0693-4bcc-bc88-8aaf7d82f424', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-1028-gcp', java.version: '1.8.0_191'
[info] Driver info: org.scalawebtest.core.browser.SeleniumFirefox$CleanedPageSourceFirefoxDriver
[info] Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 67.0.1, javascriptEnabled: true, moz:accessibilityChecks: false, moz:buildID: 20190529130856, moz:geckodriverVersion: 0.24.0, moz:headless: true, moz:processID: 5203, moz:profile: /tmp/rust_mozprofile.vikkzN..., moz:shutdownTimeout: 60000, moz:useNonSpecCompliantPointerOrigin: false, moz:webdriverClick: true, pageLoadStrategy: normal, platform: LINUX, platformName: LINUX, platformVersion: 4.15.0-1028-gcp, rotatable: false, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
[info] Session ID: adae5d6e-965e-4d67-9311-71ec8b431a22
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info]   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[info]   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info]   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[info]   at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:187)
[info]   at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:122)
[info]   at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
[info]   at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
[info]   at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:83)
[info]   at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)

[DaniRey]

As Firefox isn't stable yet on Travis CI, we will not add it to 3.0.0.

For now, everyone interested may just grab the implementation from https://github.com/unic/ScalaWebTest/tree/feature/add_firefox

As far as we can tell now, the problem is not in our implementation, but rather in our configuration of the build agent on Travis.

Adding Firefox is from now on tracked in #85