Provide a way to automate running unit tests that does not require Selenium or SauceLabs

[tomwayson]

I'd like to be able to run my unit tests continuously in multiple browsers as I modify my code, the way I can w/ Karma, and by that I mean w/o having to run a local standalone selenium server or to upload our code to SauceLabs in order to do so.

For a concise example of why I think it's a bad idea to require selenium to automate running the unit tests, see this PR:

xsokev/Dojo-Bootstrap#150

1. I wanted to update to Intern 2.1 (Upgrade to Intern 2.1 xsokev/Dojo-Bootstrap#144)
2. In the process, I noticed that tests no longer ran in FF (Tests not running in FireFox xsokev/Dojo-Bootstrap#145) due to a FF driver bug introduced when evergreen FF updated itself to v 32
3. That bug was fixed w/ an update to Selenium 2.43, but that caused the grunt task I used to launch Selenium not to work, so I swapped that out w/ another one
4. In the process of testing that I noticed that tests stopped running in Safari - now it looks like there's an issue w/ Safari driver after updating Safari

Add all that to the fact that I still have never gotten the IE driver to run on my PC, and you can see why I want to limit my dependence on Selenium to only what I absolutely need it for - functional tests.

I recognize that none of these issues are w/ the Intern itself, but rather w/ it's dependency on Selenium to automate running the unit tests. I know from looking at other issues in this repo and all threads on the Selenium forum that I'm not only person that has these issues w/ Selenium and it's launchers and browser drivers.

Re: Sauce Labs, I can't ever imagine that being the only place I run my unit tests, as they are so intertwined w/ my development process. I need them to run in a couple of seconds, not minutes, so that's a not a solution for this issue.

I guess I could wire up livereload to continuously reload client.html, but it would be nice if the Intern shipped w/ a way for me to achieve what I get w/ Karma w/o having to do all that myself for each project.

[csnover]

Hi Tom,

Thanks for writing!

It sounds like there are actually two things you want here. One, a server other than Selenium for instantiating browsers on different platforms, and two, the ability to automatically reload/rerun all your tests whenever some of your code changes.

1. “Don’t use Selenium”

I think it's a bad idea to require selenium to automate running the unit tests

Intern supports any service, device, or platform that implements the WebDriver JSON wire protocol[1][2]. It does not mandate the use of Selenium nor Sauce Labs; if you don’t like these products, you can choose or write an alternative server that implements the WebDriver HTTP endpoints, and Intern will be able to use it.

More pertinently, Intern doesn’t control where or how browsers are started; it only requests environments, and does so using the standard WebDriver API. We have no plans to implement an incompatible API for managing browser instances since there is already wide industry support for the WebDriver API, and the WebDriver API already covers this functionality (with much broader support for different environments than we’d ever be able to maintain ourselves).

a FF driver bug introduced when evergreen FF updated itself to v 32
there's an issue w/ Safari driver after updating Safari
I still have never gotten the IE driver to run on my PC

As browser vendors continue to introduce native WebDriver implementations as the Chromium team, IE team, and Mozilla team are doing, compatibility issues such as these will cease to exist since their WebDriver servers will be kept up-to-date with new browser releases. I’m sorry that things are so frequently broken upstream. We don’t have the manpower to make this any better.

2. “Automatically re-run my tests”

it would be nice if the Intern shipped w/ a way for me to achieve what I get w/ Karma w/o having to do all that myself for each project.

OK. I don’t use TDD and neither does SitePen so Intern’s internals are not designed to support this mode of operation. It would be very difficult right now to do this without simply using an external Grunt watch task, so my recommendation would be to use an external Grunt watch task to execute intern-runner when files change (or use Karma to load Intern’s client.html with a custom reporter, I guess). I plan to refactor client.js/runner.js to be much less procedurally driven in Intern 4.0, at which point it will be much, much easier to implement rapid reload functionality directly within Intern.

It’s not likely that we will implement either of these requests any time soon, given the above, but we would accept proposals/patches/financial sponsorship of the feature if you want to see it done.

Thanks again!

[tomwayson]

Thanks @csnover for your very insightful response.

Good to know where you guys are coming from, and what's on the roadmap.

It seems the Intern is just not well suited to the mode of unit testing that I'm used to. As this is not a way that you guys test, I wouldn't expect you to implement it w/o popular demand. I'm sorry to admit that I don't really understand how Karma launches and controls the browsers w/o Web Driver - but that's also the beauty of it, I don't have to know. It just works. If I did, or if I had time to research it, I definitely would have opened a PR instead of an issue.

I think the Intern is a great framework, and I sincerely wish I could make better use of it. The first time I saw a coverage report that unified the results of both functional and unit tests, I was sold. Unfortunately, ever since then I've had nothing but trouble w/ the browser drivers. I hope, as you say, that will be fixed when the browser vendors introduce native drivers that work reliably.

Thanks again.

[mightyiam]

Apart from the specific issues that @tomwayson had with some selenium versions, I can honestly reiterate both his original report and his response almost word for word.

@csnover thank you for your response.

I absolutely love the way Karma launches browsers and reruns my tests on detected changes.

I can set up livereload quite easily but is there any way that I would get console output for the tests on my terminal?

[dylans]

Reading through https://karma-runner.github.io/0.12/intro/how-it-works.html and https://github.com/karma-runner/karma/raw/master/thesis.pdf (the university thesis behind karma), basically this is what they are doing via a Node.js server:

• Serve tests to browsers within a test runner
• Collect test results from clients
• Open web browsers and run tests
• Automatically run tests again if files change within a project
• If browser is already open and running tests, tell the browser to rerun the tests when files change (uses WebSockets to communicate to the open clients)
• Made attempts to solve a few other problems along the way (DI, mocks, etc.)

Perhaps a few shortcuts to get some of the things that Intern doesn't do today would be as follows, and perhaps make this easier for someone in the community to contribute to (as it could probably just be done as a few grunt tasks with the existing code, and there are already well established watch tasks with Grunt: https://github.com/gruntjs/grunt-contrib-watch ):

• Grunt task to watch for changed files and run unit tests in all browsers listed within your Intern config by opening the browsers, and running intern, e.g. http://localhost:9001/__intern/client.html?config=foo
• Grunt task to watch for changed files and run unit+functional tests in all browsers listed within your Intern config by running intern-runner
• While useful, lower priority because there are so many other tools to do this already: A new reporter that can collect and aggregate tests results from many tests runs over time, rather than for just a single test run, so you can see regressions over time

[mightyiam]

Thank you, @dylans.
I'm not sure whether your suggestions include in some way the printing of test reports to the console.

[dylans]

I'm not sure whether your suggestions include in some way the printing of test reports to the console.

One way I see to make that work with pure unit tests is to log the results to the server, which then returns those results to the console tab where the grunt task is running. I don't see a way to simply have a task to open a browser and return console results to it, without logging the results somewhere first.

[mightyiam]

Thank you, @dylans.

[sholladay]

Presumably you are looking for something like this? https://github.com/benderjs/browser-launcher2

I wonder if there would be a non-terrible way of doing this mostly in user-land, with only minor tweaks on a fork of Intern, as a proof of concept. That theory is based on the fact that, from what I've seen, Intern doesn't make too many assumptions about how the session gets started (for unit tests).

[jason0x43]

There is now some support for this with intern-cli. You can run intern serve -o to start the Intern proxy server and automatically open the system-default browser to the client.html test page.

[indolering]

I came here with a similar request, to run unit tests in real a real browser in the background.

Coming from Polymer and their excellent WebComponentTester, I really enjoyed the ability to run unit tests in the browser. Intern could match the UX of running unit tests in Node by controlling a Selenium standalone server running the (upcoming) headless version of Chrome.

I agree that moving away from Selenium would be a bad idea for all of the reasons outlined above. Based on the summary about Karma's internals, the problem isn't with Selenium per se, but its focus on end-to-end functional testing. If anything, we should be pushing upstream Selenium to provide similar features.

[mightyiam]

This issue inspired me write a little post just now:
https://medium.com/@mightyiam/why-in-browser-unit-testing-is-declining-c0a7b14f60bf#.loyksxqwg

[dylans]

As noted earlier by @jason0x43, intern-cli now helps quite a bit with this. Also, Intern's SeleniumTunnel will now make sure you have a local copy of all of the Selenium Driver dependencies that you need to run your tests. So I think we're just about there with regards to the use case of the original request, with the exception of having a watch task to auto test things (either rerun all tests, or rerun tests only for tests related to modules that have changed).

[indolering]

Firefox and Chrome have both announced headless modes. Supporting this in the future should be as simple as altering the command used to launch Chrome.

Lock thread?

[jason0x43]

Headless mode is going to be great, but it won't eliminate the need for Selenium. Selenium isn't just used for functional tests, it's what enables Intern to start up and manage browsers. Intern-cli provides a very basic method for starting browsers without Selenium, but it's fairly limited. For more complete Selenium-free functionality, we would need to add a new tunnel type to Dig Dug, or make a new plugin entirely, that can manage browsers without Selenium.

[jason0x43]

I'm closing this in favor of https://github.com/theintern/digdug/issues/54 (Dig Dug is where this feature will eventually be implemented).