Make ScalaWebTest compatible with all implementations of Selenium WebDriver

[DaniRey]

If we do not depend on HtmlUnits HTML parsing, we can use other Drivers as well and even extract the gauge logic completely. Ideally, we can compile this for Scala.js and use it in other tools like TestCafé. This will not be possible, as we have to parse HTML (see next comment). Most probably a rewrite for JS is the better option, if we want to use Gauges in TestCafé.

[DaniRey]

This is a challenging task because:

• org.openqa.selenium.WebElement is very limited. There is no way to query its parent, siblings or children (no DOM traversal)
• WebDriver does not provide access to Response Headers and Response Code

Because of this, a few intermediate steps are needed.

• not being able to use com.gargoylesoftware.htmlunit.html.HtmlElement means we have to parse the HTML in ScalaWebTest
• not many options exist to parse HTML. So far JSoup seems to be the best option
• to get a hold of Response Headers and Response Code we have to send all requests through a proxy, which is under our control. We can then ask the proxy to get this information. (inspiration https://github.com/wkeeling/selenium-wire)

[DaniRey]

I think the final solution should provide three different kinds of Browsers

• HtmlUnitExtended (default)
• Selenium
• ProxiedSelenium

All three should implement a common interface, but similar to JDBC drivers, some methods might not be implemented, pointing to alternatives which implement said method.

HtmlUnitExtended: This webdriver is headless. It uses HtmlUnit and doesn't hide that fact.

No system prerequisites	Response Codes	Response Headers	Screenshots	JavaScript
Yes	Yes	Yes	No	Limited (RhinoJs)

Selenium: This might be Chrome, Firefox,... controlled via SeleniumWebDriver API

No system prerequisites	Response Codes	Response Headers	Screenshots	JavaScript
Browser and WebDriver installed	No	No	Yes	Yes

Selenium Proxied: Again Chrome, Firefox... controlled via SeleniumWebDriver API and using a proxy, which is under our control. This allows ScalaWebTest to read the response code and response headers. This information is not exposed by the Selenium WebDriver API and never will (see SeleniumHQ/selenium-google-code-issue-archive#141 (comment)).

No system prerequisites	Response Codes	Response Headers	Screenshots	JavaScript
Browser and WebDriver installed and Proxy running	Yes	Yes	Yes	Yes

[DaniRey]

Following the reasoning from ashley.leyba in SeleniumHQ/selenium-google-code-issue-archive#141 (comment)

If we attempt to make WebDriver the silver bullet of web testing, then it will suffer in overall quality.

1. driver.get(url) blocks until the browser has loaded the page. In the case of a login redirect, what status/headers would you want returned? By the semantics of the API, we would return the response for the final loaded page. You would end up with a 200 instead of the 302 you're looking for.

3. If you want to test HTTP request/response headers, then write an appropriately sized test and test those directly.

We should probably not add Selenium Proxied to ScalaWebTest. I can currently not think of any reasonable use case, where one would require to use Chrome/Firefox/... to test a response code or response header. For those tests, HtmlUnitExtended should be enough. Therefore I would concentrate our efforts on HtmlUnitExtended and Selenium. The ProxiedSelenium variant might follow as a module, but I doubt we will ever need it.

[DaniRey]

We should try to use or integrate with https://github.com/bonigarcia/webdrivermanager