Combining coverage from 2 test runs?

[tmayr]

I'm currently running tests in two different app settings (we have different tests depending on which country the application is launched).

I'd love to have one coverage with both runs combined as the code goes through different parts on each run.

Is this something remotely possible to do?
What would be the best practice in this scenarios?

[cpojer]

@DmitriiAbramov do we have anything for this? I think this could be pretty handy if we decide to kill our internal runner.

[aaronabramov]

it's actually pretty easy to do with istanbul as long as you have a .json coverage report.
we do something similar to this here:
https://github.com/facebook/jest/blob/master/scripts/mapCoverage.js

[ardok]

Just in case someone needs step by step to do this (I literally combined my coverage report after finally stumbled upon here).

1. My original test script was

    "test": "jest --config=src/test/jest.node.json && jest --config=src/test/jest.browser.json && npm run lint"

I also do the code coverage config thing from jest

2. I changed my config on each to have

  "coverageReporters": [],

3. I used the mapCoverage file (just copy pasted it to my project) and changed my test script to be:

    "test": "jest --config=src/test/jest.node.json && jest --config=src/test/jest.browser.json && node ./scripts/map-coverage.js && npm run lint"

Of course, don't forget to install the dependencies that you need for mapCoverage.

[tmayr]

Finally tackled this today.

Our use case was a bit different, as files tested were exactly the same, under exactly the same paths but took different code paths due to environment variables being set and running the application in different country environments.

package.json | jest.config.json

"coverageReporters": ["json"] // otherwise you don't have any coverage to read`

And we had to make some changes to our test command too, there is probably a better way but a quick 5s google search did not yield any results on how to generate coverages with different file names

package.json

"test": "jest && cp ./coverage/coverage-final.json ./coverage/coverage-chile-final.json && env COUNTRY=peru jest && cp ./coverage/coverage-final.json ./coverage/coverage-peru-final.json && node ./lib/mapCoverage.js"

mapCoverage.js had to have some changes done too, since we're reading from two coverage files with the same paths, instead of one with different paths.

mapCoverage.js

const createReporter = require('istanbul-api').createReporter;
const istanbulCoverage = require('istanbul-lib-coverage');

const map = istanbulCoverage.createCoverageMap();
const reporter = createReporter();

const countries = ['chile', 'peru'];

countries.forEach(country => {
  const coverage = require(`../coverage/coverage-${country}-final.json`);
  Object.keys(coverage).forEach(
    filename => map.addFileCoverage(coverage[filename])
  );
});

reporter.addAll(['json', 'lcov', 'text']);
reporter.write(map);

@DmitriiAbramov @cpojer Thanks for the help!
@ardok Thanks for dropping in with your experience!

[cpojer]

@DmitriiAbramov shall we make mapCoverage into a jest-map-coverage package so that it can be re-used?

[aaronabramov]

@cpojer might be a good idea. Istanbul already provides everything needed for that, but it's not documented and requires some additional configuration.

were you thinking about something like this?

const {mergeCoverage} = require('jest-map-coverage');

merge(
  'file1.json',
  'file2.json',
).replacePaths(path => path.toLowerCase());

[ljharb]

I use https://www.npmjs.com/package/istanbul-merge to merge istanbul reports - with that and #2861, it should be trivial.

[bchenSyd]

thanks @tmayr for sharing. that works for me!

actually istanbul has new api, like below

 var istanbul = require('istanbul'),
        collector = new istanbul.Collector(),
        reporter = new istanbul.Reporter(),
        sync = false;

    const countries = ['chile', 'peru'];

countries.forEach(country => {
  const coverage = require(`../coverage/coverage-${country}-final.json`);
  collector.add(coverage);
});
   
    reporter.addAll([ 'text', 'lcov', 'clover' ]);
    reporter.write(collector, sync, function () {
        console.log('All reports generated');
    });

Above code should work. but I"m getting Cannot read property 'text' of undefined which I believe is a defect in istanbul

[fahrenq]

For everyone who struggles with these nowadays and receives errors like

        Object.keys(second.s).forEach(function (k) {
               ^

TypeError: Cannot convert undefined or null to object

or

        throw new Error('Invalid file coverage object, missing keys, found:' +
        ^
Error: Invalid file coverage object, missing keys, found:data

For some reason Istanbul tooling not working with jest-generated coverage JSONs because of some weird "data" key that appears in some entries of coverage JSONs files.

I.e. these entries in the same coverage json.

"<...>/src/app.js": {"path":"<...>/src/app.js","statementMap":{"0":{"start"...
"<...>/src/aws.js": {"data":{"path":"<...>/src/aws.js","statementMap":{"0":{"start"...

I solved this by just deleting this "data" key and passing its value directly to the file-path key.

Here's my full 'mapCoverage' function.

/* eslint-disable import/no-extraneous-dependencies */
const libCoverage = require('istanbul-lib-coverage');
const { createReporter } = require('istanbul-api');

const integrationCoverage = require('../coverage-integration/coverage-final.json');
const unitCoverage = require('../coverage-unit/coverage-final.json');

const normalizeJestCoverage = (obj) => {
  const result = obj;
  Object.entries(result).forEach(([k, v]) => {
    if (v.data) result[k] = v.data;
  });
  return result;
};

const map = libCoverage.createCoverageMap();
map.merge(normalizeJestCoverage(integrationCoverage));
map.merge(normalizeJestCoverage(unitCoverage));

const reporter = createReporter();
reporter.addAll(['json', 'lcov', 'text']);
reporter.write(map);

I assume that whole Lerna powered Istanbul repository is the latest version, so, why doesn't it assume coverage files to have 'data' key? And what this 'data' key is? Why it only appears in some entries?

@cpojer if you have a little bit of time, can you please give any comments? Maybe there's a newer version of istanbul exists that respects this "data" key or something? At least I couldn't find anything. Thank you very much!

[mummybot]

We had this issue as well, digging into where the data attribute was coming from it was a typings.d.ts file to allow importing of .json modules:

// typings.d.ts
declare module '*.json' {
    const value: any;
    export default value;
}

Removing this file, and updating the Typescript ^2.9.2 tsconfig to import JSON fixed our issues

"resolveJsonModule": true,
"esModuleInterop": true,

https://stackoverflow.com/a/50674344/158815

[Morikko]

FYI: nyc report does it natively while you put all your coverage.json in the same folder.

More explanation on this other issue message.

[samboylett]

If you're using istanbul, the CLI seems to automatically merge any JSON files you use for you:

istanbul report html --include=*.coverage.out.json

With that my map coverage command just needs to clean up the JSON from Jest as described by @fahrenq

const normalizeJestCoverage = (obj) => {
    const result = { ...obj };

    Object.entries(result).forEach(([k, v]) => {
        if (v.data) result[k] = v.data;
    });

    return result;
};

const integrationCoverage = normalizeJestCoverage(require('./jest.integration.out.json').coverageMap);
const unitCoverage = normalizeJestCoverage(require('./jest.unit.out.json').coverageMap);

fs.writeFileSync('./unit.coverage.out.json', JSON.stringify(unitCoverage, null, 4));
fs.writeFileSync('./integration.coverage.out.json', JSON.stringify(integrationCoverage, null, 4));

[kirillgroshkov]

Just to summarize. I've used different suggestions here and there and this is the final script that works for me as of today that allows to combine 2 coverage reports in json format into one combined report (with set of formats: json, lcov).

/*

yarn tsn-script ./scripts/mergeCoverage.ts --report ./coverage0/coverage-final.json --report ./coverage1/coverage-final.json

*/

import * as fs from 'fs-extra'
import * as yargs from 'yargs'
const { createCoverageMap } = require('istanbul-lib-coverage')
const { createReporter } = require('istanbul-api');

main().catch(err => {
  console.error(err)
  process.exit(1)
})

async function main () {
  const argv = yargs
    .options({
      report: {
        type: 'array', // array of string
        desc: 'Path of json coverage report file',
        demandOption: true,
      },
      reporters: {
        type: 'array',
        default: ['json', 'lcov'],
      }
    })
    .argv

  const reportFiles = argv.report as string[]
  const reporters = argv.reporters as string[]

  const map = createCoverageMap({})

  reportFiles.forEach(file => {
    const r = fs.readJsonSync(file)
    map.merge(r)
  })

  const reporter = createReporter();
  // reporter.addAll(['json', 'lcov', 'text']);
  // reporter.addAll(['json', 'lcov']);
  reporter.addAll(reporters)
  reporter.write(map)
  console.log('Created a merged coverage report in ./coverage')
}

[James-Smith83]

I noticed that it can be pretty difficult to try and merge multiple coverage data from various tests. It is clear that some people have issues with trying to do this as well.
If you want to see an alternative way of doing this rather than using the "isntabul" tool then this article talks about a different methods of how to merge coverage data from multiple test runs.

https://www.rapitasystems.com/blog/merging-coverage-data-multiple-test-runs

[ochemli]

How do you check-coverage after generating a merged report?

[OshriBa]

@ochemli

https://stackoverflow.com/questions/46709413/how-to-convert-existing-istanbul-json-file-to-html

[rivy]

One of the issues I've run into (solved through some unknown means by CodeCov, see "Merging Reports") when trying to combine runs from different machines is differing path prefixes (especially *nix vs Windows machines).

I don't know of a simple way to combine the outputs under that differing path prefix circumstance. And without that merge, the coverage metric will continually flip-flip between wildly incorrect values.

[ochemli]

Thanks @OshriBa, @rivy. I was asking about enforcing coverage thresholds on a merged report. I was able to get merged coverage from jest + mocha + newman tests with the help of this thread. However, running nyc check-coverage on the merged report failed with
Error: Invalid file coverage object, missing keys
Turns out nyc will checks all coverage*.json file, including coverage-summary.json.
I solved this by moving coverage-summary.json before running nyc check-coverage

[mcblum]

@fahrenq hey did you figure this out within Lerna? We have Angular 1 along side Angular 8 and jest, and we're getting that weird data attr.

[fahrenq]

@mcblum Sorry, I completely forgot context as by now. Good luck :)

[mcblum]

All good, thank you!

[hongbo-miao]

Since istanbul is deprecated: https://www.npmjs.com/package/istanbul
I am ending up with the way using nyc by this answer at Stack Overflow: https://stackoverflow.com/a/63008134/2000548

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.