Support for alternate browsers

[sheltonial]

Would be great to have the option of running tests in an alternate browser such as Chrome. Our application has reached a point where we have so many tests that PhantomJs crashes. Seems that many others are having this issue with PhantomJs also and the solution that has worked for us is to run via Chrome.

[ArtemGovorov]

We have had the same issue with one of our apps as well.
Though wallaby.js had solved it because it does the initial run in parallel, so there are much less tests to run in each individual PhantomJs instance.

Have you already encountered PhantomJs crashes when using wallaby.js for your tests?

Anyway, we do plan to support Chrome runner at some point.

[sheltonial]

Interesting, perhaps it's an issue with our tests. I'll look into it some more. Seems to be hangining around the the 1650th test. Turning debug on within the wallaby config reveals the following output:

Wed, 04 Mar 2015 00:04:56 GMT project Sandbox [rvgd1] is responsive, closing it
Wed, 04 Mar 2015 00:04:59 GMT project Test executed: should have "..." on scope
Wed, 04 Mar 2015 00:04:59 GMT project Test executed: should have "..." on scope
Wed, 04 Mar 2015 00:04:59 GMT project Test executed: should have "..." on scope
Wed, 04 Mar 2015 00:05:01 GMT project Signal killed phantomjs #1: null, exit code: 3221225477

[ArtemGovorov]

I have set up wallaby to run angular.js project tests (around ~3500), works ok. But I know there's an issue with phantomJs that you're talking about.

Before we add the support for Chrome runner, there's one thing you can try (apart from trying to find out whether there's something can be done with the tests).

As I have said, wallaby is using multiple instances of PhantomJs to run tests, especially initially. So, if you try to run your tests using more parallel instances initially - it may help. Have a look into http://wallabyjs.com/docs/config/workers.html and try to increase workers.initial setting, and then re-running wallaby. If your tests are not in a single huge file, it may help.

[sheltonial]

Increasing the initial workers has definitely helped. I suspect it could be a memory issue with Phantom processors as if I decrease the number of initial workers to 1 it fails much sooner. Tests are split up over approx 200 files. I'm finding now it completes the tests but fails merging tests results. Output is:

Wed, 04 Mar 2015 00:47:20 GMT project Run 143 test(s), skipped 0 test(s)
Wed, 04 Mar 2015 00:47:20 GMT project Sandbox [4mtc9] is responsive, closing it
Wed, 04 Mar 2015 00:47:20 GMT project Merging parallel test run results
Finished executing: 1839 tests
Wed, 04 Mar 2015 00:47:32 GMT project Test run finished
Wed, 04 Mar 2015 00:48:18 GMT project Failed to process test run results Invalid string length
Failed to process test run results Invalid string length

[ArtemGovorov]

Nice, we are getting somewhere. Did you set "instrument: false" for libraries and other large files (patterns), that you don't really edit much and don't need coverage for, in your files list? Not only it should speed up the tests execution, but significantly decreases the memory footprint.

[sheltonial]

Setting instrument to false for library files combined with increasing the number of initial workers has solved the problem. Thanks Artem

[ArtemGovorov]

Awesome, the issue with "Invalid string length" happens when wallaby tries to JSON.stringify a huge object with coverage to send it to IDE. Excluding libraries makes the object way more compact, as coverage for them is not collected anymore. You may probably also observe some test execution speed improvement.

If you don't mind, I'll keep the issue opened, as the title and the request for other browsers support is valid.

[jamesshore]

+1 for supporting alternate browsers (and better yet, multiple browsers; and better still, integrating with Karma).

[elgalu]

+1 for Chrome support, PhantomJS is buggy while using Protractor

[vlazar]

+1 for supporting multiple real browsers even if tests are not instant but "just" much faster, than say in karma

[cResults]

+1 for supporting multiple real browser chrome, IE, FF, etc. I agree with vlazar, the results of this doesn't need to be instant. Although they are rare, karma has pointed out tests that pass on many browsers, but fail on one (usually ie), so this is unfortunately a necessary requirement for us.

[mdoleh]

+1

[quantuminformation]

Is there a way to debug tests when using the wallabyjs runner?

[ArtemGovorov]

@quantuminformation Not yet - #74

[quantuminformation]

@ArtemGovorov Ok np, I can still have an alternative runner for when I need to do this, eg Karma.

[psulat]

+1 For Chrome. I need a browser that supports webgl.

[ArtemGovorov]

Happy to announce Electron runner support. It allows you to run your tests using the latest Chromium/V8 without spinning up the browser.

[boneskull]

(pasting from #674)

I want to test Mocha itself with Wallaby.

Problems:

• Mocha needs to run itself. This has already been addressed in testing mocha with mocha #641 (thanks!)
• Mocha's matrix is somewhat large. We test against:
  • Node.js @ v0.8, v0.10, v0.12, iojs-v3.x, 4.x, 5.x, and 6.x
  • Browser @ PhantomJS 1.x, IE8, Chrome (latest), Firefox (latest), Edge (latest), Safari (latest)
• The entire suite of tests for any given platform cannot run in the same instance of Mocha.
  • The root cause is architectural limitations; you cannot run test suite A with the bdd interface and test suite B with the qunit interface with the same instance.
  • These are split into many different make targets. Running make test runs all of the targets.
  • We only get bootstrap() to choose the interface. Is there another function we can call do to this configuration per-file?
• A browser cannot execute a large portion of the integration tests, because they are built around the CLI.

So I'm thinking what I want is basically more granular control over how Wallaby runs--I should be able to specify a glob (or individual files) to run in Node.js, another glob to run in the Browser, a third to run in both. Since we're bundling with Browserify, we need to be able to do that, too,

I realize launching a browser other than PhantomJS or Electron is not supported. Chromium should have a headless mode sometime this decade, so I imagine that could be on the table eventually. But, is there a architectural reason why it isn't possible to actually launch Firefox? Is it a performance concern? It begs the question where the responsibilities of Wallaby end.

I am also having trouble envisioning a UI where the coverage for Node.js and the coverage for a browser could be simultaneously displayed inline. For me, inline coverage isn't Wallaby's killer feature--it's the instantaneous execution of changed tests (I realize others may have a different opinion).

---

(See #675 for ideas about browser-based display of information.)
(See #89 for multiple context support; though it's not clear to me if this implies "multiple run configurations" in a JetBrains environment)

[ArtemGovorov]

Google Chrome test runner support has been added (including headless mode for supported browser versions).

[smcenlly]

Closing this feature request. We have a separate feature request for firefox and we will create browser-specific feature requests as we see demand for other browsers.