Merging coverage files doesn't produce accurate cumulative result

[stahloss]

I execute the following nyc commands, after generating individual coverage reports on the same codebase using different test executions run by Jest and Cypress.

nyc merge coverage-jest combined/coverage-jest.json
nyc merge coverage-cypress combined/coverage-cypress.json
nyc report combined --reporter html --reporter text

Expected Behavior

I'd expect this to show a correct cumulative coverage report of .ts and .html files.

Observed Behavior

1. Coverage of .html files is not shown in merged report, while they are show in individial report.
2. Coverage of some .ts files is shown correctly, but for others a Cannot read property 'locations' of undefined TypeError is shown.

Troubleshooting steps

• still occurring when I put cache: false in my nyc config

Environment Information

  System:
    OS: Windows 10 10.0.18363
    CPU: (8) x64 Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
    Memory: 7.97 GB / 31.88 GB
  Binaries:
    Node: 13.6.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.0 - C:\Program Files\nodejs\yarn.CMD
    npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    istanbul-instrumenter-loader: 1.2.0 => 1.2.0
    istanbul-lib-coverage: 1.2.1 => 1.2.1
    nyc: ^15.0.1 => 15.0.1
    source-map-support: ^0.5.16 => 0.5.16
    ts-node: ^7.0.1 => 7.0.1
    typescript: ~3.7.4 => 3.7.5

For Cypress coverage I use above dependencies as well as @cypress/code-coverage:^3.0.2.
I'm using jest: 24.1.0 to produce unit test coverage.

[coreyfarrell]

This is difficult for us to support as you are not using nyc to generate any coverage. From the limited information you shared I see some version conflicts:

• istanbul-instrument-loader is using a really old version of istanbul-lib-instrument.
• jest 24 uses the previous version of all istanbul libraries
• @cypress/code-coverage@3 and nyc@15 both use the latest versions of istanbul libraries

As for your command, nyc report combined --reporter html --reporter text is incorrect. You need nyc report -t combined --reporter html --reporter text.

Further you have not linked a repository as required so we do not know how you are even running the tests, what kind of configurations you are using, etc.

[stahloss]

Thanks for your response. Yes, Jest has the istanbul instrumenter built in, so I tried to align other dependencies to that, because before the downgrade I got this error: Invalid file coverage object, missing keys, found:data. @cypress/code-coverage doesn't instrument by itself, so I can control that as opposed to Jest.

You're right the command I posted is wrong, but I used the correct one, so it's a typo.

Yes, sorry for that, I was hoping the information I supplies would be enough, because my production setup if fairly complex with a mix of nx with angular, cypress, jest and nyc. It would take some time to duplicate that setup.

I've tried to bump jest to the latest version (25.3.0) but wasn't able to merge.
From what you said I understand that it should work if I align the libraries correctly and I see that @jest/reporters@25.3.0 is using istanbul-lib-coverage@3.0.0 so if I align that with my cypress instrumentation, it should work.

I'll try to make it work with jest@25.3.0, if the problems persists I'll make a demo project or else I'll close the issue.

EDIT
I've tried with the following dependencies:

  npmPackages:
    istanbul-instrumenter-loader: 3.0.1 => 3.0.1
    istanbul-lib-coverage: 3.0.0 => 3.0.0
    nyc: ^15.0.1 => 15.0.1
    source-map-support: ^0.5.16 => 0.5.16
    ts-node: ^7.0.1 => 7.0.1
    typescript: ~3.7.4 => 3.7.5

And jest@25.3.0. This resulted in Invalid file coverage object, missing keys, found:data. Jest's coverage-final.json indeed contains data properties. Does this relate to #1226?

I've replaced istanbul-instrumenter-loader with babel-plugin-istanbul, so the outdated istanbul-lib-instrument is gone. The coverage files still cannot be merged with the one Jest produces.

[lirantal]

@D0rmouse the reason you're still getting this error Invalid file coverage object, missing keys, found:data is because you aren't producing and merging the correct reporter file.

Make sure these happen:

1. Specify the json reporter. If you're using jest it's something like this for its config:

"coverageReporters": [
      "lcov",
      "text",
      "json"
    ]

The important part being that there's a json entry there in the array

2. Then after you run the tests you'll have a coverage-final.json created in the coverage dir. That's the file you need to copy to the combined directory along with the other file of the same name from a different directory. So both of these files should exist in a directory to combine them (give them different names)

3. Then run the nyc merge and report commands

Hope it helps 🤞
Goodluck!

[stahloss]

@lirantal I do have a json entry in my coverageReporters array. I got json from the start it's just not merging correctly because the formats don't match.

[lirantal]

@D0rmouse can you confirm that you have that file called coverage-final.json created for both of your coverage report runs? Copy them to a directory, and then run the merge on that when they are both there alone with no other files.

[stahloss]

@lirantal Yes I can, the output of the Jest test is simply different than my Cypress coverage output.
I created an issue in the Jest repo as well. Can't reproduce using a minimal setup unfortunately.
jestjs/jest#9808

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[quisido]

I'm experiencing this same problem.

My coverage results look like so:

cypress/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 6, "1": 4, "2": 2, "3": 4, "4": 2, "5": 2, "6": 2 },
    "f": { "0": 4, "1": 2 },
    "b": { "0": [0, 4], "1": [2] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

jest/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "1": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "2": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "3": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "4": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "5": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 1, "1": 1, "2": 1, "3": 1, "4": 0, "5": 1 },
    "f": { "0": 1 },
    "b": { "0": [1] }
  }
}

The expected outcome would be path/to/file.ts having 100% coverage, since the gaps on one test are covered by the other test. However, instead, what you see is the gaps being reported:

combined/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 7, "1": 4, "2": 3, "3": 4, "4": 3, "5": 3, "6": 3 },
    "f": { "0": 4, "1": 3 },
    "b": { "0": [0, 4], "1": [3] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

The "coverage" merge is really just concatenated instead of merging.

It appears that: line 1 characters 1-10 uncovered + line 1 characters 1-5 covered + line 1 characters 5-10 covered = line 1 characters 1-10 uncovered (wrong) + line 1 characters 1-10 covered (right).

The uncovered characters are still shown as uncovered even though their character-span is covered by a different part of the map. Just guessing.

Here is a nice visual LCOV report for the above JSON:

Cypress (fully covered except for 1 branch):

Jest (specifically covering the above branch):

Combined report (expected to be fully covered):

Any advice to unblock this? I don't want to have to write a JSON merger manually. 😓

Maybe this is just a weird issue with TypeScript.

Below, how are imports and exports not covered? The export statement isn't covered, but the inside of the function somehow is. The inside references an import, but the import isn't covered. Nonsensical. 😵

[nishantchapa]

I am facing the same issue as the above user @CharlesStover
Any update on how to fix this issue?

[quisido]

My issue ended up simply being that the version of Jest needed to be downgraded to 26. No idea why 27 is broken. The instrumentation of Jest 27 is simply off and doesn't match the one generated by react-scripts when running the local application in dev mode for Cypress.

[nishantchapa]

I am trying to integrate this to existing application which is using Jest 27.0.6 and can't downgrade to lower version.
Any other way to fix this?

[quisido]

I spent a long time trying to fix it, and was not able to solve it with Jest 27. A downgrade to 26 resolved it like magic. This may need to be a bug report for the Jest team, as it appears 27 is not instrumenting coverage correctly, producing incorrect results for TypeScript files (this occurs with either ts-jest or Babel for transpilation). Things like "import type" should not be showing as uncovered, ever.

If you want to open a ticket with the Jest team, I think that's the right solution, and you could link to my screenshots and data in this issue as a reference for them.