Add support for Selenium end-to-end test debugging

[jtlapp]

We should be able to use one test runner for all of our testing and debugging. I'm now looking at mods that might help make subtap useful for testing and debugging with Selenium.

[jtlapp]

v0.6.1 supports running Selenium tests, but it still needs support for debugging these tests.

[jtlapp]

I spent several days trying to make it possible to debug Selenium tests in tap or subtap using the webdriver.io API. Apparently it isn't possible to reach the debugger except through the wdio test runner that webdriver.io provides, and then one can only pause execution, not step through it. So this library doesn't seem compatible with general use of a debugger.

Also, stdout or stderr written from within webdriver.io tasks gets lost under tap. I don't know whether this is a tap issue or a webdriver.io issue, but it's certainly an incompatibility between them.

So I'll focus on having subtap support debugging the Selenium webdriver API.

UPDATE: I didn't take this route, as I found ways to resolve both issues. I'm focusing on the more user-friendly library, webdriver.io.

[jtlapp]

Update! I'm able to get subtap to work with webdriver.io API. Cool features coming. You can even pause tests that you resume by hitting ENTER.

[jtlapp]

v0.7.0 now works with webdriver.io. It includes a feature that allows you to have the test wait for subtap to prompt the user before proceeding with the test. The debug() command of webdriver.io allows you to do this with wdio. I'll be releasing a separate webdriver.io library with a command that prompts subtap, but the spec describes how anyone can do this.

There were also a few issues associated with preserving stdout/stderr when a webdriver.io test errors, and those have been fixed in subtap v0.7.0.

I'm leaving this issue open for now, until I'm sure that I've got all the basics covered.

[jtlapp]

When hitting multiple browsers and servers at the backend, there's all sorts of opportunities for tests to timeout. If a timeout happens in the middle of a lengthy test file and the run terminates as a result, you have to rerun the whole series of subtests to try again. This should be unnecessary.

It seems that we need a way to time out individual root subtests without timing out the whole test run. When a root subtest times out, subsequent root subtests should be able to run, because test numbering won't be affected.

Interesting that the tap test runner only has a per-file timeout. That's not working with Selenium, where tests can run for minutes. subtap is using an inactivity timeout -- each time the file provides information, I know it's still functioning, so I reset the timeout.

UPDATE: Because all subtests run in the same thread, it seems that the best I could do is cause the next test to begin running and terminate the timed-out test the next time it tests an assertion. I couldn't prevent the prior test from suddenly resuming in the middle of the subsequent test, perhaps with negative consequences for the subsequent test.

UPDATE #2: However, I could completely terminate the test file and then rerun it starting from the next subtest. I could also specify a number of retries for timeouts and keep rerunning the file from the timed-out subtest until either there's no timeout or we reach the retry count.

[jtlapp]

Implementing retries requires being able to close open browser windows on timeout. My initial attempts to do this via node-cleanup are not working. I'll have to look into interactions among tap, subtap, and webdriverio to resolve this.