Mocha stack traces make testing unusable #391

Closed
twpayne opened this Issue Mar 18, 2013 · 12 comments

Comments

Projects
None yet
3 participants
@twpayne
Contributor

twpayne commented Mar 18, 2013

Mocha is currently unusable for Test Driven Development.

There are several problems:

  • Stack traces are only shown in the browser window, not in the console. Therefore, there is no way to click on the filename / line number to jump to the line which is failing. Instead, you have to manually navigate through the source code to set a breakpoint on an appropriate line. This is insane.
  • Mocha disables horizontal scrolling and stack traces are truncated: if your screen is not large enough it is impossible to see the filename or line number.
  • Mocha stops at the first failing test. This makes it impossible to know which other tests pass or not.

If we want contributors to submit tests in ol3, then we have to provide a usable testing framework. As we've already changed testing framework a few times from JSUnit (ugly but it worked), to Jasmine (fashionable at the time but abandoned by upstream within a few months), to now Mocha/sinon/expect.js (fashionable, pretty, fragile, un-extensible, and unusable), perhaps someone would care to suggest workarounds for the above or suggest a new bandwagon?

mocha-is-useless

@Turbo87

This comment has been minimized.

Show comment
Hide comment
@Turbo87

Turbo87 Mar 18, 2013

Contributor

I'll have a look.

Contributor

Turbo87 commented Mar 18, 2013

I'll have a look.

@tschaub

This comment has been minimized.

Show comment
Hide comment
@tschaub

tschaub Mar 18, 2013

Member

In my opinion, Mocha is an improvement over Jasmine for the async support. The HTML runner is not an improvement, but most don't focus on the HTML runner. I'd like to put together a pull request to use Testacular. This would encourage testing in multiple browsers and would focus on a console runner. Using Phantom also focusses people away from the HTML runner.

If we want an HTML runner that suits our needs, I suggest we work on one. But I don't think Test Driven Development necessarily means a swanky HTML test runner.

Member

tschaub commented Mar 18, 2013

In my opinion, Mocha is an improvement over Jasmine for the async support. The HTML runner is not an improvement, but most don't focus on the HTML runner. I'd like to put together a pull request to use Testacular. This would encourage testing in multiple browsers and would focus on a console runner. Using Phantom also focusses people away from the HTML runner.

If we want an HTML runner that suits our needs, I suggest we work on one. But I don't think Test Driven Development necessarily means a swanky HTML test runner.

@Turbo87

This comment has been minimized.

Show comment
Hide comment
@Turbo87

Turbo87 Mar 18, 2013

Contributor
  • Stack traces are only shown in the browser window, not in the console. Therefore, there is no way to click on the filename / line number to jump to the line which is failing. Instead, you have to manually navigate through the source code to set a breakpoint on an appropriate line. This is insane.

this is what I get after running ./build.py test:

turbo TMacBookV: -Code-ol3_002

  • Mocha disables horizontal scrolling and stack traces are truncated: if your screen is not large enough it is impossible to see the filename or line number.

seems to be some weird CSS issue, that I haven't fully understood yet...

  • Mocha stops at the first failing test. This makes it impossible to know which other tests pass or not.

fixed in #394

#374

we could look into http://chaijs.com/ instead of expect.js. I don't have any experience with that framework either, but it seems that it could be configured to support the same syntax as expect.js. It seems to have a plugin function and support for sinon is already provided by such a plugin (http://chaijs.com/plugins/sinon-chai).

if everyone agrees I could have a look into replacing expect.js with chai.js.

Contributor

Turbo87 commented Mar 18, 2013

  • Stack traces are only shown in the browser window, not in the console. Therefore, there is no way to click on the filename / line number to jump to the line which is failing. Instead, you have to manually navigate through the source code to set a breakpoint on an appropriate line. This is insane.

this is what I get after running ./build.py test:

turbo TMacBookV: -Code-ol3_002

  • Mocha disables horizontal scrolling and stack traces are truncated: if your screen is not large enough it is impossible to see the filename or line number.

seems to be some weird CSS issue, that I haven't fully understood yet...

  • Mocha stops at the first failing test. This makes it impossible to know which other tests pass or not.

fixed in #394

#374

we could look into http://chaijs.com/ instead of expect.js. I don't have any experience with that framework either, but it seems that it could be configured to support the same syntax as expect.js. It seems to have a plugin function and support for sinon is already provided by such a plugin (http://chaijs.com/plugins/sinon-chai).

if everyone agrees I could have a look into replacing expect.js with chai.js.

@twpayne

This comment has been minimized.

Show comment
Hide comment
@twpayne

twpayne Mar 18, 2013

Contributor

Thanks so much for your Herculean efforts on this @Turbo87!

By "Stack traces are only shown in the browser window, not in the console." I should have clarified that I meant the browser's dev tools console, not the command line output of ./build.py.

The HTML runner is not an improvement, but most don't focus on the HTML runner.

I think the HTML runner is essential if you hope to use the browser's dev tools to help you develop code. When you're debugging you don't care how pretty it looks when your tests pass, but you do care very much how much the testing framework helps or hinders you when your tests fails.

Contributor

twpayne commented Mar 18, 2013

Thanks so much for your Herculean efforts on this @Turbo87!

By "Stack traces are only shown in the browser window, not in the console." I should have clarified that I meant the browser's dev tools console, not the command line output of ./build.py.

The HTML runner is not an improvement, but most don't focus on the HTML runner.

I think the HTML runner is essential if you hope to use the browser's dev tools to help you develop code. When you're debugging you don't care how pretty it looks when your tests pass, but you do care very much how much the testing framework helps or hinders you when your tests fails.

@Turbo87

This comment has been minimized.

Show comment
Hide comment
@Turbo87

Turbo87 Mar 18, 2013

Contributor

you could have a look into https://github.com/eeroan/WebConsole-reporter but I don't know whether that will disable the normal HTML reporter...

Contributor

Turbo87 commented Mar 18, 2013

you could have a look into https://github.com/eeroan/WebConsole-reporter but I don't know whether that will disable the normal HTML reporter...

@twpayne

This comment has been minimized.

Show comment
Hide comment
@twpayne

twpayne Mar 18, 2013

Contributor

we could look into http://chaijs.com/ instead of expect.js. I don't have any experience with that framework either, but it seems that it could be configured to support the same syntax as expect.js. It seems to have a plugin function and support for sinon is already provided by such a plugin (http://chaijs.com/plugins/sinon-chai).

An initial scan of the Chai documentation looks promising. It supports the concept of "deep" equality which (I assume) means that you can compare different instances of arrays and objects for equality (and so remove the equalArray function added in #374). It also seems to be more extensible, but I have not yet looked in more detail.

Let's make sure our next testing infrastructure meets our requirements before switching to it. Here's an initial quick list.

  • Can run tests from the command line / Travis.
  • Can run tests in the browser.
  • Provides easy-to-use stack traces when tests fail in the browser.
  • Testing continues even after several failures.
  • Provides fundamental testing functionality (e.g. c9f43b4)
  • Can be easily extended to include ol3-specific tests (e.g. 93577bd)
  • (Wishlist) Can run tests in parallel, like JSUnit
  • (Wishlist) Does not silently pass when there are trivial problems (#375)
  • (Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

Please add more as required.

Contributor

twpayne commented Mar 18, 2013

we could look into http://chaijs.com/ instead of expect.js. I don't have any experience with that framework either, but it seems that it could be configured to support the same syntax as expect.js. It seems to have a plugin function and support for sinon is already provided by such a plugin (http://chaijs.com/plugins/sinon-chai).

An initial scan of the Chai documentation looks promising. It supports the concept of "deep" equality which (I assume) means that you can compare different instances of arrays and objects for equality (and so remove the equalArray function added in #374). It also seems to be more extensible, but I have not yet looked in more detail.

Let's make sure our next testing infrastructure meets our requirements before switching to it. Here's an initial quick list.

  • Can run tests from the command line / Travis.
  • Can run tests in the browser.
  • Provides easy-to-use stack traces when tests fail in the browser.
  • Testing continues even after several failures.
  • Provides fundamental testing functionality (e.g. c9f43b4)
  • Can be easily extended to include ol3-specific tests (e.g. 93577bd)
  • (Wishlist) Can run tests in parallel, like JSUnit
  • (Wishlist) Does not silently pass when there are trivial problems (#375)
  • (Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

Please add more as required.

@Turbo87

This comment has been minimized.

Show comment
Hide comment
@Turbo87

Turbo87 Mar 18, 2013

Contributor
  • (Wishlist) Can run tests in parallel, like JSUnit

see https://groups.google.com/forum/?fromgroups=#!topic/mochajs/47uqR1Eakbo

  • (Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

This is possible in Mocha by clicking on the relevant test suite. It will append e.g. ?grep=ol.parser... to the URL and will only focus on that suite then. I haven't tried appending that parameter to the URL that phantomjs is calling, maybe that will work too.

Contributor

Turbo87 commented Mar 18, 2013

  • (Wishlist) Can run tests in parallel, like JSUnit

see https://groups.google.com/forum/?fromgroups=#!topic/mochajs/47uqR1Eakbo

  • (Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

This is possible in Mocha by clicking on the relevant test suite. It will append e.g. ?grep=ol.parser... to the URL and will only focus on that suite then. I haven't tried appending that parameter to the URL that phantomjs is calling, maybe that will work too.

@twpayne

This comment has been minimized.

Show comment
Hide comment
@twpayne

twpayne Mar 18, 2013

Contributor

(Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

This is possible in Mocha by clicking on the relevant test suite. It will append e.g. ?grep=ol.parser... to the URL and will only focus on that suite then. I haven't tried appending that parameter to the URL that phantomjs is calling, maybe that will work too.

I'd like to be a bit more clever. For example, pretty much everything depends indirectly on ol.Coordinate. An modification to ol.Coordinate to trigger a lot of tests to run, even those without ol.Coordinate in the it() description.

Contributor

twpayne commented Mar 18, 2013

(Wishlist) Ability to run a sub-set of the tests when only a few source code files have changed (e.g. by cleverly parsing the goog.provide / goog.require dependency graph)

This is possible in Mocha by clicking on the relevant test suite. It will append e.g. ?grep=ol.parser... to the URL and will only focus on that suite then. I haven't tried appending that parameter to the URL that phantomjs is calling, maybe that will work too.

I'd like to be a bit more clever. For example, pretty much everything depends indirectly on ol.Coordinate. An modification to ol.Coordinate to trigger a lot of tests to run, even those without ol.Coordinate in the it() description.

@twpayne

This comment has been minimized.

Show comment
Hide comment
@twpayne

twpayne Mar 18, 2013

Contributor

The HTML runner is not an improvement, but most don't focus on the HTML runner.

I'd like to put together a pull request to use Testacular.

Just to clarify, using Testacular requires a decent HTML runner, right?

Contributor

twpayne commented Mar 18, 2013

The HTML runner is not an improvement, but most don't focus on the HTML runner.

I'd like to put together a pull request to use Testacular.

Just to clarify, using Testacular requires a decent HTML runner, right?

@tschaub

This comment has been minimized.

Show comment
Hide comment
@tschaub

tschaub Mar 18, 2013

Member

Just to clarify, using Testacular requires a decent HTML runner, right?

I'll finish the pull request and you can take a look. It doesn't use the HTML runner or our ol.html markup. You can just keep a web inspector console open and click on line numbers from stack traces when there are failures.

Member

tschaub commented Mar 18, 2013

Just to clarify, using Testacular requires a decent HTML runner, right?

I'll finish the pull request and you can take a look. It doesn't use the HTML runner or our ol.html markup. You can just keep a web inspector console open and click on line numbers from stack traces when there are failures.

@Turbo87

This comment has been minimized.

Show comment
Hide comment
@Turbo87

Turbo87 Mar 19, 2013

Contributor

if it's just that then I guess you could also run mocha directly if you have node installed... I just used the redirection through phantomjs because I did not want increase the number of dependencies for now.

Contributor

Turbo87 commented Mar 19, 2013

if it's just that then I guess you could also run mocha directly if you have node installed... I just used the redirection through phantomjs because I did not want increase the number of dependencies for now.

@twpayne

This comment has been minimized.

Show comment
Hide comment
@twpayne

twpayne Oct 10, 2013

Contributor

This is effectively resolved thanks to @Turbo87 with f0df0d7.

Contributor

twpayne commented Oct 10, 2013

This is effectively resolved thanks to @Turbo87 with f0df0d7.

@twpayne twpayne closed this Oct 10, 2013

@juliemr juliemr referenced this issue in angular/protractor Apr 11, 2014

Closed

Switch off Stacktrace output #696

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment