Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Support code coverage / istanbul #229

Open
Raynos opened this Issue · 34 comments
@Raynos

I've been using istanbul recently for server-side stuff and it's really nice.

Doing code coverage for client side requires a server to send client-side generate coverage reports to and then using server side tooling to convert coverage reports (json objects) into lcov or html.

Having this functionality in testem would be really nice.

@airportyh
Owner

Yup yup. On my radar. I thought for server-side node-cover already does a great job?

@Raynos

I've found istanbul to be nicer for my workflow ( https://github.com/Raynos/hash-router/blob/master/package.json#L38 ). I've also configured it to send data to coveralls (like travis but for coverage) on my CI tests ( https://github.com/Raynos/hash-router/blob/master/package.json#L37 )

@webpro

+1 Code coverage reporting would be a absolutely great :)

@searls

+0.5

@BenDavidJamin

+1 I'd like to know what I forgot to test. Or at least know where the holes are.

@davemo

+0.75

@Raynos

karma has a coverage extension ( https://github.com/karma-runner/karma-coverage ).

There is a blog post for coverage ( http://ariya.ofilabs.com/2013/10/code-coverage-of-jasmine-tests-using-istanbul-and-karma.html )

Not sure how karma handles coverage files PER browser.

@airportyh
Owner

@Raynos thanks for the info!

@terinjokes

Another support for code coverage, was surprised to find this wasn't already something supported.

@airportyh
Owner

Okay, okay, I am listening.

@jasonpincin

+1 hacking this together with before_tests and a custom launcher, less hack would be great

@rwjblue rwjblue referenced this issue in stefanpenner/ember-app-kit
Merged

replaced karma with testem #292

@webpro

I've got some code coverage working, so I thought to just share it here.

https://gist.github.com/webpro/7644285

  • Gruntfile.js contains configuration for the testem task, but could also be in a testem.json file.
  • If you have Istanbul installed as a local dependency, just replace istanbul with node_modules/istanbul/lib/cli.js.
  • The runner.html file contains a solution when using Require.js + Jasmine. The point is that the instrumented source files should be used, and when the tests are done, the window.__coverage__ data is POSTed, and you catch it over there:
  • The collect-coverage.js is some local server that simply saves any data it receives to coverage.json. Within the Testem Grunt task it's possible to use functions; but this could be anything you can start from CLI.
@Raynos

getting browserify support for code coverage is going to take a bit more machinery. maybe running istanbul on the test entry point.

@jasonpincin

@Raynos Currently I'm running istanbul to generate covered sources from lib to lib-cov, then browserifying the test entry point which includes from lib-cov (rewriting lib to lib-cov in a browserify transform). I do this in a before_tests step. All we need at that point is a way to collect the window.__coverage__ data.

@Raynos

I don't have the lib / lib-cov set up. I would have to do something similar with cov-index and cov-some-folder and have all the paths rewritten.

I think the easiest thing is to actually browserify the code and then have istanbul generate a covered version of the bundle but that would mean istanbul needs to understand the source file mapping, maybe read source maps.

@jasonpincin

I had brought this up (kind of) with @gotwarlost in gotwarlost/istanbul#59 a while back, and he's the one that suggested the instrument-then-browserify approach.

The advantage of instrumenting first, is you get to pick which files you want to report coverage on. If you were to browserify the test entry point, and then have istanbul cover the bundle, it would be generating coverage for non-implementation files (such as the tests themselves). So even with support for source maps, you'd need to go one step further and be able to "filter" out coverage who's source mapped file didn't match some pattern, otherwise your coverage numbers would be off.

That said, I agree it would be nice to be able to do this with less machinery and build steps. The process can be slow, making for less-than-immedate test feedback when developing, and that lag can be bothersome. If istanbul could "understand" browserify packages, we could potentially use something like watchify which offers more efficient re-bundles I believe.

@davemo

It may be useful to look at @searls contributions to node-sandboxed-module that added code coverage support: felixge/node-sandboxed-module@3d323ee

@searls

Naive question: anyone tried blanket.js for coverage? Setup story looks way easier than istanbul/JSCov for getting coverage of browser-land tests http://blanketjs.org

@airportyh
Owner

Started looking into this. Here's a DIY example https://github.com/airportyh/testem/tree/master/examples/coverage_istanbul

@Raynos

@airportyh that's one approach, but it relies on this magical src folder and instrumented folder.

That approach doesn't work with other projects that are based around npm / browserify conventions.

@Raynos

Really nice!

@gotwarlost

+1. Very nicely done.

@terinjokes

What if we aren't using browserify?

@Raynos

you can try the testem-grunt approach that works purely based on files and not on browserify

@terinjokes

@Raynos: No, i meant with istanbulify, it's a browserify transform.

@Raynos

Well istanbulify solves the browserify specific problem which file system transforms don't solve, its a really nice solution to a problem browserify created :D

@airportyh
Owner

It would be nice to have a coverage plugin that is composable with a browserify plugin or a coffeescript plugin. I have been wracking by brain over this, but I don't think this is possible in practice.

@cliren

We are using testem from https://www.npmjs.org/package/grunt-contrib-testem. Is there a way to integrate istanbul or other coverage tools into it so that a grunt task can genrerate coverage on ci tools? I posted an issue here inossidabile/grunt-contrib-testem#26 to see if grunt-contrib-testem team can help us with that.

@danactive

I too would use test coverage tools for testem via https://www.npmjs.org/package/grunt-contrib-testem

@piuccio

@danactive and @cliren have you tried grunt-testem yet? It's not contrib but it does the job and has code coverage.

@amercier

+1 For integration within Testem. Code instrumentation + reporting should be handled by a Testem "plugin". Is there a way to hook a code "preprocessor" (istanbul instrument) and a "reporter" (istanbul report) as in Karma?

It shouldn't need any Grunt/Gulp-like tools for that. Karma doesn't.

Also, coverage.json generated by Istanbul should never be written to disk, it should stay in memory (ie the "plugin" should use Istanbul's node.js API, not the command-line API).

@jehoshua02

+1 For :birthday: :gift:?

@jehoshua02 jehoshua02 referenced this issue in jehoshua02/testem-example
Open

coverage #8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.