Use with code coverage / Istanbul?

[neopostmodern]

I am trying to get code coverage with the following command

istanbul cover ./node_modules/mocha-webpack/bin/mocha-webpack --report lcovonly -- -R spec

But that only covers my webpack config files! Because coverage/coverage.json only says {".../webpack.config.js":{...

(I copied this straight from https://github.com/nickmerwin/node-coveralls#istanbul and replaced mocha with mocha-webpack.)

[zinserjan]

To be honest, didn't tried code coverage with mocha-webpack yet...

Maybe it's just enough to apply a loader for code coverage (see istanbul-instrumenter-loader).

[hikirsch]

+1

[ntdb]

I got this working! I used babel-plugin-istanbul for instrumentation and nyc for reporting. Some of this is specific to my setup but here are the key snippets:

In package.json:

{
  "devDependencies": {
    "babel-plugin-istanbul": "^1.1.0",
    "nyc": "^7.1.0",
    ...
  },
  "scripts": {
    "test": "NODE_ENV=test nyc mocha-webpack test/**/*.{js,jsx}"
  },
  "nyc": {
    "reporter": [
      "lcov",
      "text-summary"
    ],
    "exclude": [
      ".tmp",
      "test"
    ]
  }
}

In .babelrc:

{
  "env": {
    "test": {
      "plugins": ["istanbul"]
    }
  }
}

[ThaNarie]

I needed to include "node_modules" in the "exclude" array to have it run without errors, worked perfectly after that!

FYI, the NODE_ENV=test part doesn't work cross-platform, the cross-env npm module can be used for that.

[ThaNarie]

Since we're talking about coverage, I do have a question.

At the moment, it only shows coverage for the components that are referenced in the test file, so the coverage results gives a wrong view when looking at the complete codebase.
Ideally you want to add coverage for all the source files, which means they should be added to webpack as well.

I guess it could be done by passing "(test/**/*.spec.js|src/app/component/**/index.js)" as entry to mocha-webpack, but that slows things down a bit since the globParent is now the root of the project and will traverse all other folders including node_modules as well.
And it doesn't look that straight-forward to me.

I also tried to hack in another approach that was more in line the karma-webpack setup I use in other projects, which is to create a second context for the source files.

In the prepareEntry.js that would mean adding the last 3 lines with the source path:

  return "\n" +
    "var testsContext = require.context(\"" + path + "\", false);\n" +
    "console.log(testsContext.keys())\n" +
    "testsContext.keys().forEach(testsContext);\n" +
    "\n" +
    "var sourceContext = require.context(\"" + "../../src/app/component" + "\", false);\n" +
    "console.log(sourceContext.keys())\n" +
    "sourceContext.keys().forEach(sourceContext);\n";

And adding another contextReplacementPlugin in prepareWebpack.js:

// add another matcher, parent, and update matchModule to use them, and add:
webpackPlugins.push((0, _contextReplacementPlugin2.default)('../../src/app/component', matchModule, recursive));

I guess sourcePath could be added as an extra CLI option?

How do you feel about these approaches? Is it worth starting a PR for this?

[zinserjan]

@ThaNarie this is not only an issue for code coverage reports. So let's move conversation to #45 for this.

[serut]

This is what I use (typescript + webpack + react + mocha-webpack) :

npm install --save-dev istanbul-instrumenter-loader nyc

nyc mocha-webpack --webpack-config webpack.coverage.config.js \"(tests|web_modules)/**/*.test.\\?(ts|tsx)\" --require source-map-support/register

webpack config:

  // Enable sourcemaps for debugging webpack's output.
  devtool: "inline-source-map",
  module: {
    loaders: [
      {
        test: /\.tsx{0,1}?$/,
        exclude: [/node_modules/, /json/],
        loaders: ['istanbul-instrumenter', "babel-loader", "ts-loader"]
      }
    ]
  }

  plugins: [
    // Allow to define React as a global variable for JSX/TSX.
    new webpack.ProvidePlugin({"React": "react",}),
  ]

.bablerc

{
  "presets": [
    "es2015",
    "react"
  ],
}

[nshoes]

@ntdb thanks that worked for me!

[jennylia]

@serut It worked for me by testing it on the commandline.

[serut]

I'm still using istanbul-instrumenter-loader, basically it works, but that's not so great, I write ES6 code and I get ES5 code coverage. I tryed to use https://github.com/istanbuljs/babel-plugin-istanbul with no success.

EDITED: See 6 comments below my working configuration with babel-plugin-istanbul

Here is my setup:
webpack.conf.js:

[...],
devtool: 'source-map',
module: {
    loaders: [{
       test: /\.jsx?$/,
       exclude: [/node_modules/, /json/],
       loader: 'babel',
    }
}
plugins: [
    new webpack.DefinePlugin({
      'process.env': {
        NODE_ENV: JSON.stringify('coverage'),
      },
   }),
],

.babelrc

{
  "presets": [
    "es2015", "react", "stage-3"
  ],
  "plugins": ["transform-class-properties", "add-module-exports"],
  "env": {
    "coverage": {
      "plugins": ["istanbul"]
    }
  }
}

.nycrc

{
  "require": [
    "babel-core/register",
    "source-map-support/register"
  ],
  "extension": [
    ".jsx",
    ".js"
  ],
  "reporter": [
    "lcov",
    "html"
  ],
  "sourceMap": false,
  "instrument": false,
  "report-dir": "./reports/coverage"
}

And the command I execute:
"test:coverage": "NODE_ENV=coverage nyc --show-process-tree mocha-webpack --webpack-config webpack.conf.js \"/src/**/*.test.js{,x}\" --recursive --require source-map-support/register",

🐐

[zinserjan]

Don't know if this is related, but webpack had an issue with sourcemaps parsing of babel (webpack/source-list-map#3).

You can either make sure that all of your dependencies are using source-list-map in version 0.0.7 or just install webpack 1.14 (that should bump everything else automatically).

[Freddixx]

I have the very same problem. Using webpack 1.14 and mocha-webpack 0.7.0 all I see is coverage for my webpack config. Coverage runs on nyc and babel-plugin-istanbul.

[zinserjan]

I tried code coverage today with the latest master and everything works fine for me with just nyc and istanbul-instrumenter-loader. Tried it with .ts & .js (babel) files.

I'll write some docs for this setup that I used in my test repo: mocha-webpack-example. If someone wants to try that out, the repo relies on a linked/latest version of mocha-webpack.

@serut
Try istanbul-instrumenter-loader in version 1.2.0. They added better sourcemap support, which should fix your ES5 code coverage issues.

[Freddixx]

Jan, that repo was a huge help. I managed to set everything up. Strange enough it worked with the instrumenter and not with the babel-plugin-istanbul even though the latter has much more traction on npm.

Thanks for the quick help!

[zinserjan]

Babel-plugin-istanbul works also for me. I used that before I included code coverage for typescript. See this commit zinserjan/mocha-webpack-example@f1af8be.

[mikeengland]

@zinserjan This worked for me, thank you for the example commit!!

[serut]

Ok it works really great with babel-plugin-istanbul :
.bablerc:

{
  "presets": [
    "es2015", "react", "stage-3"
  ],
  "plugins": ["transform-class-properties", "add-module-exports"],
  "env": {
    "development": {
      "plugins": ["react-hot-loader/babel"]
    },
    "coverage": {
      "plugins": ["istanbul"]
    }
  }
}

.nycrc:

{
  "extension": [
    ".jsx",
    ".js"
  ],
  "exclude": [
    "test",
    "test{,-*}.js",
    "**/*.test.js",
    "**/__tests__/**",
    "**/node_modules/**",
    "**/tests/**"
  ],
  "reporter": [
    "lcov",
    "html",
    "text-summary"
  ],
  "cache": true,
  "sourceMap": false,
  "instrument": false,
  "report-dir": "./reports/coverage"
}

webpack.coverage.config.js:

const webpack = require('webpack')
const nodeExternals = require('webpack-node-externals')


// Setup babel coverage environment
process.env.NODE_ENV = 'coverage'

let config = {
  target: 'node', // in order to ignore built-in modules like path, fs, etc.
  externals: [nodeExternals({
   // whitelist: [], <-- depends of your project
  })], // in order to ignore all modules in node_modules folder
  // Enable sourcemaps for debugging webpack's output.
  devtool: 'source-map',
  verbose: true,
  displayErrorDetails: true,
  stats: {
    colors: true,
    reasons: true,
  },
  module: {
    noParse: [
      /sinon/,
      /iconv-loader/,
      /enzyme/,
    ],
    loaders: [
      // Transpile ES6 Javascript into ES5 with babel loader
      {
        test: /\.jsx?$/,
        exclude: [/node_modules/, /json/],
        loader: 'babel',
      },
   ]
  },
  // enable sourcemaps support
  output: {
    devtoolModuleFilenameTemplate: '[absolute-resource-path]',
    devtoolFallbackModuleFilenameTemplate: '[absolute-resource-path]?[hash]',
  },
}

module.exports = config

Finally, my run command in the package.json (ready for large project) :

"test:coverage": "node --max_old_space_size=4096 ./node_modules/.bin/nyc node --max_old_space_size=4096 ./node_modules/.bin/mocha-webpack --timeout 30000 --webpack-config webpack.coverage.config.js \"@(tests|web_modules)/**/*.test.js{,x}\"",

To sum up, the only difference with my previous test is process.env.NODE_ENV = 'coverage' inside the webpack.coverage.config.js file

[Izhaki]

I've managed to get this to work

• with: webpack, mocha, mocha-webpack, nyc
• without: babel, ts-node, bells and whistles

See this comment for how

This is very similar to @zinserjan example

[JesterXL]

I tried @ntdb 's example, @zinserjan 's example, and @Izhaki 's and all result in 0% coverage. It's running, just not finding my files for some reason.

[Izhaki]

@JesterXL Can you try the setup mentioned here?

[JesterXL]

@Izhaki You rock, thanks, but I figured it out. I basically uninstalled everything, and used:

Just mocha, no mocha-webpack, or any of it's ilk.

package.json scripts looks like this:

"test": "mocha --compilers js:babel-core/register --require babel-polyfill src/**/*.test.js",
"coverage": "NODE_ENV=test nyc --reporter=lcov --reporter=text mocha --compilers js:babel-core/register --require babel-polyfill 'src/**/*.test.js'",

Made sure .babelrc had the env:

"env": {
    "test": {
      "plugins": [ "istanbul" ]
    }
  }

Nothing required for the webpack like all the 20 billion sites say, but here it is all the same:

module.exports = {
  entry: ['babel-polyfill', './src/index.js'],
  devtool: "inline-source-map",
  output: {
    filename: 'bundle.js'
  }
};

I should point out, too, that this is a front-end JavaScript project using Pixi, but not all my tests deal with DOM and whatnot, so I was afraid I'd have to whip out the karma, but so far, no need. The npm test is fast, and npm run coverage is a bit slower, but still fast compared to Karma, so I'm happy to finally get it working!

[Izhaki]

Well. I haven't quite delve into this, but...

mocha-webpack is a webpack plugin and it works on files within the webpack 'file system', which are not written to disk.

Any coverage tool, unless a webpack plugin as well, should not marry well with any other webpack plugin.

This is why you need mocha and not mocha-webpack to get the coverage working.

There's more to say, but that's the gist of it.

[JesterXL]

So should I change my setup to use mocha-webpack again, or...?

[Izhaki]

@JesterXL On the contrary - stick to mocha for code coverage.

[modosc]

here's how we got it working (nyc + mocha-webpack + babel-plugin-istanbul):
in package.json:

  "nyc": {
    "all": false,
    "extension": [
      ".js"
    ],
    "reporter": [
      "lcov",
      "cobertura"
    ],
    "exclude": [
      "**/node_modules/**",
      ".tmp",
      "test",
    ],
    "report-dir": "./test/coverage",
    "cache": true,
    "sourceMap": false,
    "instrument": false
  },

(note the sourceMap and instrument settings - these are important)
our babel config is generated by js so you'll have to figure this out on your own if you have a static one:

      if (testCoverage) {
        // add istanbul in to instrument our code - do this here instead of with
        // istanbul-instrumenter-loader because the latter seems to ignore our
        // config
        //
        // also for some reason we have to duplicate the exclude settings here or
        // else some of them don't work ('test') - but if we remove
        // the setting altogether from package.json then we get memory errors because
        // nyc tries to parse node_modules.
        const exclude = require('../package').nyc.exclude
        object.plugins.push(['istanbul', { exclude }])

we use inline-cheap-module-source-map in webpack for test coverage runs.

and then from the cli call mocha-webpack as usual with nyc --:

./node_modules/.bin/nyc -- ./node_modules/.bin/mocha-webpack ...