Skip to content

Loading…

Mocha stack traces make testing unusable #391

Closed
twpayne opened this Issue · 12 comments

3 participants

@twpayne

Mocha is currently unusable for Test Driven Development.

There are several problems:

  • Stack traces are only shown in the browser window, not in the console. Therefore, there is no way to click on the filename / line number to jump to the line which is failing. Instead, you have to manually navigate through the source code to set a breakpoint on an appropriate line. This is insane.
  • Mocha disables horizontal scrolling and stack traces are truncated: if your screen is not large enough it is impossible to see the filename or line number.
  • Mocha stops at the first failing test. This makes it impossible to know which other tests pass or not.

If we want contributors to submit tests in ol3, then we have to provide a usable testing framework. As we've already changed testing framework a few times from JSUnit (ugly but it worked), to Jasmine (fashionable at the time but abandoned by upstream within a few months), to now Mocha/sinon/expect.js (fashionable, pretty, fragile, un-extensible, and unusable), perhaps someone would care to suggest workarounds for the above or suggest a new bandwagon?

mocha-is-useless

@Turbo87

I'll have a look.

@tschaub
OpenLayers member

In my opinion, Mocha is an improvement over Jasmine for the async support. The HTML runner is not an improvement, but most don't focus on the HTML runner. I'd like to put together a pull request to use Testacular. This would encourage testing in multiple browsers and would focus on a console runner. Using Phantom also focusses people away from the HTML runner.

If we want an HTML runner that suits our needs, I suggest we work on one. But I don't think Test Driven Development necessarily means a swanky HTML test runner.

@Turbo87
  • Stack traces are only shown in the browser window, not in the console. Therefore, there is no way to click on the filename / line number to jump to the line which is failing. Instead, you have to manually navigate through the source code to set a breakpoint on an appropriate line. This is insane.

this is what I get after running ./build.py test:

turbo TMacBookV: -Code-ol3_002

  • Mocha disables horizontal scrolling and stack traces are truncated: if your screen is not large enough it is impossible to see the filename or line number.

seems to be some weird CSS issue, that I haven't fully understood yet...

  • Mocha stops at the first failing test. This makes it impossible to know which other tests pass or not.

fixed in #394

#374

we could look into http://chaijs.com/ instead of expect.js. I don't have any experience with that framework either, but it seems that it could be configured to support the same syntax as expect.js. It seems to have a plugin function and support for sinon is already provided by such a plugin (http://chaijs.com/plugins/sinon-chai).

if everyone agrees I could have a look into replacing expect.js with chai.js.

@twpayne

Thanks so much for your Herculean efforts on this @Turbo87!

By "Stack traces are only shown in the browser window, not in the console." I should have clarified that I meant the browser's dev tools console, not the command line output of ./build.py.

The HTML runner is not an improvement, but most don't focus on the HTML runner.

I think the HTML runner is essential if you hope to use the browser's dev tools to help you develop code. When you're debugging you don't care how pretty it looks when your tests pass, but you do care very much how much the testing framework helps or hinders you when your tests fails.

@Turbo87

you could have a look into https://github.com/eeroan/WebConsole-reporter but I don't know whether that will disable the normal HTML reporter...

@twpayne

we could look into http://chaijs.com/ instead of expect.js. I don't have any experience with that framework either, but it seems that it could be configured to support the same syntax as expect.js. It seems to have a plugin function and support for sinon is already provided by such a plugin (http://chaijs.com/plugins/sinon-chai).

An initial scan of the Chai documentation looks promising. It supports the concept of "deep" equality which (I assume) means that you can compare different instances of arrays and objects for equality (and so remove the equalArray function added in #374). It also seems to be more extensible, but I have not yet looked in more detail.

Let's make sure our next testing infrastructure meets our requirements before switching to it. Here's an initial quick list.

  • Can run tests from the command line / Travis.
  • Can run tests in the browser.
  • Provides easy-to-use stack traces when tests fail in the browser.
  • Testing continues even after several failures.
  • Provides fundamental testing functionality (e.g. c9f43b4)
  • Can be easily extended to include ol3-specific tests (e.g. 93577bd)
  • (Wishlist) Can run tests in parallel, like JSUnit
  • (Wishlist) Does not silently pass when there are trivial problems (#375)
  • (Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

Please add more as required.

@Turbo87
  • (Wishlist) Can run tests in parallel, like JSUnit

see https://groups.google.com/forum/?fromgroups=#!topic/mochajs/47uqR1Eakbo

  • (Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

This is possible in Mocha by clicking on the relevant test suite. It will append e.g. ?grep=ol.parser... to the URL and will only focus on that suite then. I haven't tried appending that parameter to the URL that phantomjs is calling, maybe that will work too.

@twpayne

(Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

This is possible in Mocha by clicking on the relevant test suite. It will append e.g. ?grep=ol.parser... to the URL and will only focus on that suite then. I haven't tried appending that parameter to the URL that phantomjs is calling, maybe that will work too.

I'd like to be a bit more clever. For example, pretty much everything depends indirectly on ol.Coordinate. An modification to ol.Coordinate to trigger a lot of tests to run, even those without ol.Coordinate in the it() description.

@twpayne

The HTML runner is not an improvement, but most don't focus on the HTML runner.

I'd like to put together a pull request to use Testacular.

Just to clarify, using Testacular requires a decent HTML runner, right?

@tschaub
OpenLayers member

Just to clarify, using Testacular requires a decent HTML runner, right?

I'll finish the pull request and you can take a look. It doesn't use the HTML runner or our ol.html markup. You can just keep a web inspector console open and click on line numbers from stack traces when there are failures.

@Turbo87

if it's just that then I guess you could also run mocha directly if you have node installed... I just used the redirection through phantomjs because I did not want increase the number of dependencies for now.

@twpayne

This is effectively resolved thanks to @Turbo87 with f0df0d7.

@twpayne twpayne closed this
@juliemr juliemr referenced this issue in angular/protractor
Closed

Switch off Stacktrace output #696

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.