Support code coverage / istanbul

[Raynos]

I've been using istanbul recently for server-side stuff and it's really nice.

Doing code coverage for client side requires a server to send client-side generate coverage reports to and then using server side tooling to convert coverage reports (json objects) into lcov or html.

Having this functionality in testem would be really nice.

[airportyh]

Yup yup. On my radar. I thought for server-side node-cover already does a great job?

[Raynos]

I've found istanbul to be nicer for my workflow ( https://github.com/Raynos/hash-router/blob/master/package.json#L38 ). I've also configured it to send data to coveralls (like travis but for coverage) on my CI tests ( https://github.com/Raynos/hash-router/blob/master/package.json#L37 )

[webpro]

+1 Code coverage reporting would be a absolutely great :)

[searls]

+0.5

[BenDavidJamin]

+1 I'd like to know what I forgot to test. Or at least know where the holes are.

[davemo]

+0.75

[Raynos]

karma has a coverage extension ( https://github.com/karma-runner/karma-coverage ).

There is a blog post for coverage ( http://ariya.ofilabs.com/2013/10/code-coverage-of-jasmine-tests-using-istanbul-and-karma.html )

Not sure how karma handles coverage files PER browser.

[airportyh]

@Raynos thanks for the info!

[terinjokes]

Another support for code coverage, was surprised to find this wasn't already something supported.

[airportyh]

Okay, okay, I am listening.

[jasonpincin]

+1 hacking this together with before_tests and a custom launcher, less hack would be great

[webpro]

I've got some code coverage working, so I thought to just share it here.

https://gist.github.com/webpro/7644285

• Gruntfile.js contains configuration for the testem task, but could also be in a testem.json file.
• If you have Istanbul installed as a local dependency, just replace istanbul with node_modules/istanbul/lib/cli.js.
• The runner.html file contains a solution when using Require.js + Jasmine. The point is that the instrumented source files should be used, and when the tests are done, the window.__coverage__ data is POSTed, and you catch it over there:
• The collect-coverage.js is some local server that simply saves any data it receives to coverage.json. Within the Testem Grunt task it's possible to use functions; but this could be anything you can start from CLI.

[wpannell]

+1

[Raynos]

getting browserify support for code coverage is going to take a bit more machinery. maybe running istanbul on the test entry point.

[jasonpincin]

@Raynos Currently I'm running istanbul to generate covered sources from lib to lib-cov, then browserifying the test entry point which includes from lib-cov (rewriting lib to lib-cov in a browserify transform). I do this in a before_tests step. All we need at that point is a way to collect the window.__coverage__ data.

[Raynos]

I don't have the lib / lib-cov set up. I would have to do something similar with cov-index and cov-some-folder and have all the paths rewritten.

I think the easiest thing is to actually browserify the code and then have istanbul generate a covered version of the bundle but that would mean istanbul needs to understand the source file mapping, maybe read source maps.

[jasonpincin]

I had brought this up (kind of) with @gotwarlost in gotwarlost/istanbul#59 a while back, and he's the one that suggested the instrument-then-browserify approach.

The advantage of instrumenting first, is you get to pick which files you want to report coverage on. If you were to browserify the test entry point, and then have istanbul cover the bundle, it would be generating coverage for non-implementation files (such as the tests themselves). So even with support for source maps, you'd need to go one step further and be able to "filter" out coverage who's source mapped file didn't match some pattern, otherwise your coverage numbers would be off.

That said, I agree it would be nice to be able to do this with less machinery and build steps. The process can be slow, making for less-than-immedate test feedback when developing, and that lag can be bothersome. If istanbul could "understand" browserify packages, we could potentially use something like watchify which offers more efficient re-bundles I believe.

[davemo]

It may be useful to look at @searls contributions to node-sandboxed-module that added code coverage support: felixge/node-sandboxed-module@3d323ee

[searls]

Naive question: anyone tried blanket.js for coverage? Setup story looks way easier than istanbul/JSCov for getting coverage of browser-land tests http://blanketjs.org

[airportyh]

Started looking into this. Here's a DIY example https://github.com/airportyh/testem/tree/master/examples/coverage_istanbul

[Raynos]

@airportyh that's one approach, but it relies on this magical src folder and instrumented folder.

That approach doesn't work with other projects that are based around npm / browserify conventions.

[airportyh]

I found this https://github.com/fdecampredon/istanbulify

[Raynos]

Really nice!

[gotwarlost]

+1. Very nicely done.

[terinjokes]

What if we aren't using browserify?

[Raynos]

you can try the testem-grunt approach that works purely based on files and not on browserify

[terinjokes]

@Raynos: No, i meant with istanbulify, it's a browserify transform.

[Raynos]

Well istanbulify solves the browserify specific problem which file system transforms don't solve, its a really nice solution to a problem browserify created :D

[airportyh]

It would be nice to have a coverage plugin that is composable with a browserify plugin or a coffeescript plugin. I have been wracking by brain over this, but I don't think this is possible in practice.

[cliren]

We are using testem from https://www.npmjs.org/package/grunt-contrib-testem. Is there a way to integrate istanbul or other coverage tools into it so that a grunt task can genrerate coverage on ci tools? I posted an issue here inossidabile/grunt-testem-mincer#26 to see if grunt-contrib-testem team can help us with that.

[danactive]

I too would use test coverage tools for testem via https://www.npmjs.org/package/grunt-contrib-testem

[piuccio]

@danactive and @cliren have you tried grunt-testem yet? It's not contrib but it does the job and has code coverage.

[amercier]

+1 For integration within Testem. Code instrumentation + reporting should be handled by a Testem "plugin". Is there a way to hook a code "preprocessor" (istanbul instrument) and a "reporter" (istanbul report) as in Karma?

It shouldn't need any Grunt/Gulp-like tools for that. Karma doesn't.

Also, coverage.json generated by Istanbul should never be written to disk, it should stay in memory (ie the "plugin" should use Istanbul's node.js API, not the command-line API).

[jehoshua02]

+1 For 🎂 🎁?

[laland]

+1.5

[antonmedv]

+2

[benoror]

+10

[cbuteau]

So I tried the home grown istanbul example project which currently does not work out of the box. But it was a good starting point.

I figured out the actual commandline for the istanbul instrumentation luckliy my project is a single file.

node ./node_modules/istanbul/lib/cli.js instrument <filename>.js --output instrument/<filename>.js

But when I tried to run testem in dev mode it would not work because it was launching the customized coverage testem.js config file.

I brought it up in node-debug and found I could exit before_tests and after_tests if I checked the appMode...

before_tests: function(config, data, callback)
  if (config.appMode === 'dev') {
      config.fileOptions.serve_files[0] = '<filename>.js';
      return;
  } else {
     //istanbul magic
  }

This worked in that the script would not crash but it no longer launched the browsers like it used to.

If I browsed to url it would run the tests but also crash with the following error

:\Node\jsigs\node_modules\http-proxy\lib\http-proxy\index.js:119
    throw err;
    ^

Error: connect ECONNREFUSED 127.0.0.1:7358
    at Object.exports._errnoException (util.js:890:11)
    at exports._exceptionWithHostPort (util.js:913:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1059:14)

Please help I would like to run istanbul for code coverage and regular dev mode when debugging tests and my library.

Update:
I have worked around this issue by maintaining a YAML configuration for dev testing and the JS configuration for the istanbul execution.

[johanneswuerbach]

An example for code coverage is now available here https://github.com/testem/testem/tree/master/examples/coverage_istanbul