New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code Coverage #346

Open
denis-yuen opened this Issue Dec 14, 2016 · 41 comments

Comments

@denis-yuen

denis-yuen commented Dec 14, 2016

I was wondering if Cypress had a recommended tool for calculating code coverage? and if so are there any examples?

Following up on cypress-io/cypress-example-recipes#3 and opening an issue here for discussion as recommended.

For context, we previously used Protractor paired with Istanbul ( https://github.com/gotwarlost/istanbul ) to calculate code coverage for our application. We were wondering if any one has experience with something similar when working with Cypress

FYI @agduncan94

@brian-mann

This comment has been minimized.

Show comment
Hide comment
@brian-mann

brian-mann Dec 14, 2016

Member

There is nothing currently built into Cypress to do this. Adding code coverage around e2e tests is much harder than unit and its possible it may not be feasible to do in a generic way.

If you can supply the processes / code you used to add coverage via Protractor that would be helpful.

The problem is that your typical JS src files can be written in many different ways, ES6, Typescript, Coffeescript, and generally the final JS files are transpiled, and then bundled via Browserify or Webpack.

It is possible to add code coverage to JS files served by a webserver though, so it's something we might be able to solve. IE: using something like this: https://github.com/gotwarlost/istanbul-middleware

But in doing so, the problem is that it would have to be integrated at your server level, and is specific to each way you build your src files - aka each individual application.

It may be possible for Cypress to do this at the network proxy level, but it would unlikely to be easy.

Member

brian-mann commented Dec 14, 2016

There is nothing currently built into Cypress to do this. Adding code coverage around e2e tests is much harder than unit and its possible it may not be feasible to do in a generic way.

If you can supply the processes / code you used to add coverage via Protractor that would be helpful.

The problem is that your typical JS src files can be written in many different ways, ES6, Typescript, Coffeescript, and generally the final JS files are transpiled, and then bundled via Browserify or Webpack.

It is possible to add code coverage to JS files served by a webserver though, so it's something we might be able to solve. IE: using something like this: https://github.com/gotwarlost/istanbul-middleware

But in doing so, the problem is that it would have to be integrated at your server level, and is specific to each way you build your src files - aka each individual application.

It may be possible for Cypress to do this at the network proxy level, but it would unlikely to be easy.

@agduncan94

This comment has been minimized.

Show comment
Hide comment
@agduncan94

agduncan94 Dec 16, 2016

In our application we actually used Karma along with Istanbul (karma-coverage) to calculate code coverage on our unit tests.

We setup a Grunt task which uses the karma-coverage plugin to generate a coverage report on the unit tests. We have a karma config file similar to the one in the karma-coverage link above where we specify which JavaScript files to test.

What we were hoping to do as an alternate to karma and karma-coverage is to just call Istanbul and Mocha directly (or use something else if more appropriate) on the tests. This does not work because cypress has some special variables (like cy) which are not recognized by Mocha.

This is an example of how we would expect to call Istanbul and mocha if we were just using mocha directly, instead of cypress.

So I think we are most interested in coverage for unit testing.

agduncan94 commented Dec 16, 2016

In our application we actually used Karma along with Istanbul (karma-coverage) to calculate code coverage on our unit tests.

We setup a Grunt task which uses the karma-coverage plugin to generate a coverage report on the unit tests. We have a karma config file similar to the one in the karma-coverage link above where we specify which JavaScript files to test.

What we were hoping to do as an alternate to karma and karma-coverage is to just call Istanbul and Mocha directly (or use something else if more appropriate) on the tests. This does not work because cypress has some special variables (like cy) which are not recognized by Mocha.

This is an example of how we would expect to call Istanbul and mocha if we were just using mocha directly, instead of cypress.

So I think we are most interested in coverage for unit testing.

@brian-mann

This comment has been minimized.

Show comment
Hide comment
@brian-mann

brian-mann Dec 16, 2016

Member

@agduncan94 thanks for that explanation.

To clarify - doing unit testing when it comes to code coverage is fairly straightforward and absolutely can be done.

To date Cypress has been primarily used as integration and e2e testing whereby code coverage is substantially different.

With 0.18.0 we are now seeing users begin to add in unit testing and I have described the updates we need to make to Cypress to make this a first class feature..

Member

brian-mann commented Dec 16, 2016

@agduncan94 thanks for that explanation.

To clarify - doing unit testing when it comes to code coverage is fairly straightforward and absolutely can be done.

To date Cypress has been primarily used as integration and e2e testing whereby code coverage is substantially different.

With 0.18.0 we are now seeing users begin to add in unit testing and I have described the updates we need to make to Cypress to make this a first class feature..

@mike-engel

This comment has been minimized.

Show comment
Hide comment
@mike-engel

mike-engel Oct 17, 2017

This might be a bit naïve, but would it be possible to use the new code coverage tool in the chrome dev tools to accomplish this? Admittedly, I haven't looked into it much, and it would only work in chrome for now. https://developers.google.com/web/updates/2017/04/devtools-release-notes#coverage

mike-engel commented Oct 17, 2017

This might be a bit naïve, but would it be possible to use the new code coverage tool in the chrome dev tools to accomplish this? Admittedly, I haven't looked into it much, and it would only work in chrome for now. https://developers.google.com/web/updates/2017/04/devtools-release-notes#coverage

@pvdyck

This comment has been minimized.

Show comment
Hide comment
@pvdyck

pvdyck Oct 17, 2017

There should be a way to instrument the tests to extract coverage and even export it to https://coveralls.io/ ( lemurheavy/coveralls-public#1023 )

Any chance to see this happening ? What do we need ?

pvdyck commented Oct 17, 2017

There should be a way to instrument the tests to extract coverage and even export it to https://coveralls.io/ ( lemurheavy/coveralls-public#1023 )

Any chance to see this happening ? What do we need ?

@brian-mann

This comment has been minimized.

Show comment
Hide comment
@brian-mann

brian-mann Oct 17, 2017

Member

Cypress is open source, so you're welcome to take a look around to see what we'd need to do.

Code coverage during e2e tests is completely different than unit tests. There is no dependency tree, there are 3rd party libs, and there is no access to the original source (even with source maps) due to the browser build process that every JS codebase undergoes.

Cypress could theoretically instrument the code for you at the network layer, as its headed for the browser. But this has substantial challenges - it would be unidirectional, and we'd have no way of mapping the underlying built JS code back to the original (even with source maps its not enough).

So while we could provide code coverage on the final built JS files, it would be mostly garbage from babel, typescript, or whatever else you're using to transpile. It would also include all of the 3rd party code you import in.

All in all, not very useful.

We'd likely need to expose and build an API on top of Cypress's backend process that your server could send the original source files which we could map against the transpiled versions to come up with a lightweight set of instrumented files. We could then use those to determine the standard code coverage.

To be clear - this isn't a challenge as long as the code is instrumented properly, Cypress like every other tool could easily generate code coverage. The problem is that during e2e tests we receive the final built JS code where trying to go backwards to instrument is not really going to work.

Another option is that your own webserver could send the instrumented code (that you'd have to manage yourself). After you do that, it would be fairly trivial to have Cypress be able to read off the instrumented reports after test execution and then aggregate the results.

Lot of potential directions to go here, and is likely a good bit of work to champion something that's generic enough to work in most situations but easy enough so people could drop it in without writing a lot of code for their unique situation.

Member

brian-mann commented Oct 17, 2017

Cypress is open source, so you're welcome to take a look around to see what we'd need to do.

Code coverage during e2e tests is completely different than unit tests. There is no dependency tree, there are 3rd party libs, and there is no access to the original source (even with source maps) due to the browser build process that every JS codebase undergoes.

Cypress could theoretically instrument the code for you at the network layer, as its headed for the browser. But this has substantial challenges - it would be unidirectional, and we'd have no way of mapping the underlying built JS code back to the original (even with source maps its not enough).

So while we could provide code coverage on the final built JS files, it would be mostly garbage from babel, typescript, or whatever else you're using to transpile. It would also include all of the 3rd party code you import in.

All in all, not very useful.

We'd likely need to expose and build an API on top of Cypress's backend process that your server could send the original source files which we could map against the transpiled versions to come up with a lightweight set of instrumented files. We could then use those to determine the standard code coverage.

To be clear - this isn't a challenge as long as the code is instrumented properly, Cypress like every other tool could easily generate code coverage. The problem is that during e2e tests we receive the final built JS code where trying to go backwards to instrument is not really going to work.

Another option is that your own webserver could send the instrumented code (that you'd have to manage yourself). After you do that, it would be fairly trivial to have Cypress be able to read off the instrumented reports after test execution and then aggregate the results.

Lot of potential directions to go here, and is likely a good bit of work to champion something that's generic enough to work in most situations but easy enough so people could drop it in without writing a lot of code for their unique situation.

@denis-yuen

This comment has been minimized.

Show comment
Hide comment
@denis-yuen

denis-yuen Oct 17, 2017

Thanks for the explanation BTW!

denis-yuen commented Oct 17, 2017

Thanks for the explanation BTW!

@bahmutov

This comment has been minimized.

Show comment
Hide comment
@bahmutov

bahmutov Oct 17, 2017

Collaborator

@denis-yuen are you trying to compute code coverage for your tests (they should be executing all lines, so 100% right away...) or production code? If production code, maybe stick https://github.com/bahmutov/was-tested between server and Cypress and get code coverage from there - but we do not think code coverage is helpful in production E2E tests.

We are thinking about a different type of coverage - UI element coverage. You are probably more interested in knowing if all buttons or menu elements on the page were exercised by the Cypress end to end tests, rather than all code.

Collaborator

bahmutov commented Oct 17, 2017

@denis-yuen are you trying to compute code coverage for your tests (they should be executing all lines, so 100% right away...) or production code? If production code, maybe stick https://github.com/bahmutov/was-tested between server and Cypress and get code coverage from there - but we do not think code coverage is helpful in production E2E tests.

We are thinking about a different type of coverage - UI element coverage. You are probably more interested in knowing if all buttons or menu elements on the page were exercised by the Cypress end to end tests, rather than all code.

@atfzl

This comment has been minimized.

Show comment
Hide comment
@atfzl

atfzl Feb 13, 2018

I have set up coverage for Cypress if anyone wants to try.

First you need to instrument your code, for this I have used istanbul-instrumenter-loader with webpack.

Then add this code in your <project>/cypress/support/index.js

const istanbul = require('istanbul-lib-coverage');

const map = istanbul.createCoverageMap({});

Cypress.on('window:before:unload', e => {
  const coverage = e.currentTarget.__coverage__;

  if (coverage) {
    map.merge(coverage);
  }
});

after(() => {
  cy.window().then(win => {
    const coverage = win.__coverage__;

    if (coverage) {
      map.merge(coverage);
    }

    cy.writeFile('.nyc_output/out.json', JSON.stringify(map));
    cy.exec('nyc report --reporter=html');
  });
});

Here 'window:before:unload' event is used for capturing coverage which would be lost after a cy.visit() or refresh etc.

you also need to install nyc to generate reports which will be saved in <project>/coverage folder.

atfzl commented Feb 13, 2018

I have set up coverage for Cypress if anyone wants to try.

First you need to instrument your code, for this I have used istanbul-instrumenter-loader with webpack.

Then add this code in your <project>/cypress/support/index.js

const istanbul = require('istanbul-lib-coverage');

const map = istanbul.createCoverageMap({});

Cypress.on('window:before:unload', e => {
  const coverage = e.currentTarget.__coverage__;

  if (coverage) {
    map.merge(coverage);
  }
});

after(() => {
  cy.window().then(win => {
    const coverage = win.__coverage__;

    if (coverage) {
      map.merge(coverage);
    }

    cy.writeFile('.nyc_output/out.json', JSON.stringify(map));
    cy.exec('nyc report --reporter=html');
  });
});

Here 'window:before:unload' event is used for capturing coverage which would be lost after a cy.visit() or refresh etc.

you also need to install nyc to generate reports which will be saved in <project>/coverage folder.

@bahmutov

This comment has been minimized.

Show comment
Hide comment
@bahmutov

bahmutov Feb 13, 2018

Collaborator

NICE @atfzl !!!

Collaborator

bahmutov commented Feb 13, 2018

NICE @atfzl !!!

@9fingerdev

This comment has been minimized.

Show comment
Hide comment
@9fingerdev

9fingerdev Feb 14, 2018

Been looking at this but so far have not quite figured out how to get things working. Getting an out.json file but its empty. @atfzl can you shed some more light on what steps you took to config istanbul-instrumenter-loader?

9fingerdev commented Feb 14, 2018

Been looking at this but so far have not quite figured out how to get things working. Getting an out.json file but its empty. @atfzl can you shed some more light on what steps you took to config istanbul-instrumenter-loader?

@atfzl

This comment has been minimized.

Show comment
Hide comment
@atfzl

atfzl Feb 15, 2018

@9fingerdev this is where i used the istanbul-istrumenter-loader:

rules: [
      {
        test: /\.(jsx|js|tsx|ts)$/,
        include: path.resolve(__dirname, '../src'),
        rules: (isDebug
          ? [
              {
                loader: 'istanbul-instrumenter-loader',
                options: {
                  esModules: true,
                },
              },
            ]
          : []
        ).concat([{ loader: 'awesome-typescript-loader' }]),
      },

I'll create a recipe project on weekend showing complete process.

atfzl commented Feb 15, 2018

@9fingerdev this is where i used the istanbul-istrumenter-loader:

rules: [
      {
        test: /\.(jsx|js|tsx|ts)$/,
        include: path.resolve(__dirname, '../src'),
        rules: (isDebug
          ? [
              {
                loader: 'istanbul-instrumenter-loader',
                options: {
                  esModules: true,
                },
              },
            ]
          : []
        ).concat([{ loader: 'awesome-typescript-loader' }]),
      },

I'll create a recipe project on weekend showing complete process.

@9fingerdev

This comment has been minimized.

Show comment
Hide comment
@9fingerdev

9fingerdev Feb 26, 2018

@atfzl any progress on that recipe project? We are using Mithril.js + WebPack and keep running into problems.

9fingerdev commented Feb 26, 2018

@atfzl any progress on that recipe project? We are using Mithril.js + WebPack and keep running into problems.

@atfzl

This comment has been minimized.

Show comment
Hide comment
@atfzl

atfzl commented Feb 27, 2018

@atfzl

This comment has been minimized.

Show comment
Hide comment
@atfzl

atfzl Feb 27, 2018

For some reason istanbul-instrumenter-loader does not work with babel. For working with babel you can try babel-plugin-istanbul

atfzl commented Feb 27, 2018

For some reason istanbul-instrumenter-loader does not work with babel. For working with babel you can try babel-plugin-istanbul

@atfzl

This comment has been minimized.

Show comment
Hide comment
@atfzl

atfzl Feb 27, 2018

To instrument code with babel, there is no need to change the webpack config, just add the istanbul plugin.
sample .babelrc for working instrumented code:

{
  "presets": [
    "react",
    "flow",
    [
      "es2015",
      {
        "modules": false
      }
    ],
    "stage-2"
  ],
  "plugins": [
    "transform-runtime",
    "lodash",
    "transform-decorators-legacy",
    "istanbul"
  ],
  "env": {
    "test": {
      "plugins": [
        "transform-es2015-modules-commonjs"
      ]
    }
  },
  "retainLines": true
}

atfzl commented Feb 27, 2018

To instrument code with babel, there is no need to change the webpack config, just add the istanbul plugin.
sample .babelrc for working instrumented code:

{
  "presets": [
    "react",
    "flow",
    [
      "es2015",
      {
        "modules": false
      }
    ],
    "stage-2"
  ],
  "plugins": [
    "transform-runtime",
    "lodash",
    "transform-decorators-legacy",
    "istanbul"
  ],
  "env": {
    "test": {
      "plugins": [
        "transform-es2015-modules-commonjs"
      ]
    }
  },
  "retainLines": true
}
@gkemp94

This comment has been minimized.

Show comment
Hide comment
@gkemp94

gkemp94 Feb 27, 2018

@atfzl This is exactly what I was looking for. We're using gulp though. Would the configuration be similar?

gkemp94 commented Feb 27, 2018

@atfzl This is exactly what I was looking for. We're using gulp though. Would the configuration be similar?

@atfzl

This comment has been minimized.

Show comment
Hide comment
@atfzl

atfzl Feb 28, 2018

@gkemp94 if you are using babel then you only need to add plugin in .babelrc, should work with gulp as well.

atfzl commented Feb 28, 2018

@gkemp94 if you are using babel then you only need to add plugin in .babelrc, should work with gulp as well.

@abataub

This comment has been minimized.

Show comment
Hide comment
@abataub

abataub Mar 19, 2018

@atfzl Can you please highlight how can I configure it with grunt? I'm not using webpack at all. Thanks.

abataub commented Mar 19, 2018

@atfzl Can you please highlight how can I configure it with grunt? I'm not using webpack at all. Thanks.

@robinyy

This comment has been minimized.

Show comment
Hide comment
@robinyy

robinyy Apr 1, 2018

Instrumenting the codes will do ONLY with non-compressed JS. However, a typical process of shipping the normal web application would be: Linting & UT passed (source code or transpiled) -> Packaging (minified & mangled) -> E2E/Integration testing (against compressed version of JS) -> promote to production

It would be difficult to calculate the coverage in the above pipeline unless we create a separated one only for the E2E coverage

robinyy commented Apr 1, 2018

Instrumenting the codes will do ONLY with non-compressed JS. However, a typical process of shipping the normal web application would be: Linting & UT passed (source code or transpiled) -> Packaging (minified & mangled) -> E2E/Integration testing (against compressed version of JS) -> promote to production

It would be difficult to calculate the coverage in the above pipeline unless we create a separated one only for the E2E coverage

@abataub

This comment has been minimized.

Show comment
Hide comment
@abataub

abataub Jun 4, 2018

@atfzl Can you please show light on to find code coverage when running tests in isolation mode?
cypress run --spec cypress/integration/*

Cypress version: 3.0.1

abataub commented Jun 4, 2018

@atfzl Can you please show light on to find code coverage when running tests in isolation mode?
cypress run --spec cypress/integration/*

Cypress version: 3.0.1

@atfzl

This comment has been minimized.

Show comment
Hide comment
@atfzl

atfzl Jun 5, 2018

@abataub the steps i have mentioned also works in isolation mode. Let me know what is the exact issue.
And for making it work with grunt see the comment at #346 (comment) if you are using babel.

atfzl commented Jun 5, 2018

@abataub the steps i have mentioned also works in isolation mode. Let me know what is the exact issue.
And for making it work with grunt see the comment at #346 (comment) if you are using babel.

@abataub

This comment has been minimized.

Show comment
Hide comment
@abataub

abataub Jun 5, 2018

Thanks, @atfzl. I just made changes and it works.

abataub commented Jun 5, 2018

Thanks, @atfzl. I just made changes and it works.

@crosscompile

This comment has been minimized.

Show comment
Hide comment
@crosscompile

crosscompile Jul 18, 2018

Thanks @atfzl that was super helpful!

In order to end up with cumulative coverage across multiple cypress/integration/*.js files, I had to add some code to read and merge any existing '.nyc_output/out.json file. Not sure if anyone else ran into the same issue.

crosscompile commented Jul 18, 2018

Thanks @atfzl that was super helpful!

In order to end up with cumulative coverage across multiple cypress/integration/*.js files, I had to add some code to read and merge any existing '.nyc_output/out.json file. Not sure if anyone else ran into the same issue.

@afenton90

This comment has been minimized.

Show comment
Hide comment
@afenton90

afenton90 Jul 24, 2018

@atfzl have you tried this with env babel plugin? I can't get it to work in my setup.
How sensitive is the babel example you have given?

afenton90 commented Jul 24, 2018

@atfzl have you tried this with env babel plugin? I can't get it to work in my setup.
How sensitive is the babel example you have given?

@paulfalgout

This comment has been minimized.

Show comment
Hide comment
@paulfalgout

paulfalgout Aug 17, 2018

Contributor

The v3 solution I got working.. it's probably overkilling it in some form, but this allowed for the node stored map to be reused across all of the tests merging them along the way. I set the env coverage to open when running cypress open and it only runs the report for whichever file I run.

In plugins:

    if(config.env.coverage) {
        const istanbul = require('istanbul-lib-coverage');
        coverageMap = istanbul.createCoverageMap({});

        on('task', {
            'coverage'(coverage) {
                coverageMap.merge(coverage);
                return JSON.stringify(coverageMap);
            }
        });
    }

In support:

if(Cypress.env('coverage')) {
    afterEach(function() {
        const coverageFile = `${ Cypress.config('coverageFolder') }/out.json`;

        cy.window().then(win => {
            const coverage = win.__coverage__;

            if(!coverage) return;

            cy.task('coverage', coverage).then(map => {
                cy.writeFile(coverageFile, map);

                if(Cypress.env('coverage') === 'open') {
                    cy.exec('nyc report --reporter=html');
                }
            });
        });
    });
}

My npm script is essentially:
cross-env NODE_ENV=test yarn run build && cypress run --env coverage=true && nyc report --reporter=html --reporter=text

Contributor

paulfalgout commented Aug 17, 2018

The v3 solution I got working.. it's probably overkilling it in some form, but this allowed for the node stored map to be reused across all of the tests merging them along the way. I set the env coverage to open when running cypress open and it only runs the report for whichever file I run.

In plugins:

    if(config.env.coverage) {
        const istanbul = require('istanbul-lib-coverage');
        coverageMap = istanbul.createCoverageMap({});

        on('task', {
            'coverage'(coverage) {
                coverageMap.merge(coverage);
                return JSON.stringify(coverageMap);
            }
        });
    }

In support:

if(Cypress.env('coverage')) {
    afterEach(function() {
        const coverageFile = `${ Cypress.config('coverageFolder') }/out.json`;

        cy.window().then(win => {
            const coverage = win.__coverage__;

            if(!coverage) return;

            cy.task('coverage', coverage).then(map => {
                cy.writeFile(coverageFile, map);

                if(Cypress.env('coverage') === 'open') {
                    cy.exec('nyc report --reporter=html');
                }
            });
        });
    });
}

My npm script is essentially:
cross-env NODE_ENV=test yarn run build && cypress run --env coverage=true && nyc report --reporter=html --reporter=text

@hangvar

This comment has been minimized.

Show comment
Hide comment
@hangvar

hangvar Aug 23, 2018

I'm having problems with:

TS2339: Property '__coverage__' does not exist on type 'Window'.

so I'm guessing my instrumentation is not working. Since I'm running Cypress 3.1.0 it would be really interesting if @paulfalgout could share a recipe project.

hangvar commented Aug 23, 2018

I'm having problems with:

TS2339: Property '__coverage__' does not exist on type 'Window'.

so I'm guessing my instrumentation is not working. Since I'm running Cypress 3.1.0 it would be really interesting if @paulfalgout could share a recipe project.

@paulfalgout

This comment has been minimized.

Show comment
Hide comment
@paulfalgout

paulfalgout Aug 23, 2018

Contributor

@hangvar ah yep, so that code above assumes the src is instrumented. I used babel-plugin-istanbul to do so my .babelrc looks like:

{
    "presets": ["@babel/preset-env"],
    "env": {
      "test": {
        "plugins": [ "istanbul" ]
      }
    }
}

which is why I needed the cross-env NODE_ENV=test in the npm script.

However any form of instrumenting is fine, if you don't use babel, but do use webpack istanbul-instrumenter-loader will also work. instrumenting will make the __coverage__ variable attached to window as the tests are run.

The code above I added takes that __coverage__ variable and merges it together along the way, writing the merged result to a file.

Then the npm command uses nyc to make a report from that file.

I might be able to whip up a repo soon with an example.

Contributor

paulfalgout commented Aug 23, 2018

@hangvar ah yep, so that code above assumes the src is instrumented. I used babel-plugin-istanbul to do so my .babelrc looks like:

{
    "presets": ["@babel/preset-env"],
    "env": {
      "test": {
        "plugins": [ "istanbul" ]
      }
    }
}

which is why I needed the cross-env NODE_ENV=test in the npm script.

However any form of instrumenting is fine, if you don't use babel, but do use webpack istanbul-instrumenter-loader will also work. instrumenting will make the __coverage__ variable attached to window as the tests are run.

The code above I added takes that __coverage__ variable and merges it together along the way, writing the merged result to a file.

Then the npm command uses nyc to make a report from that file.

I might be able to whip up a repo soon with an example.

@hangvar

This comment has been minimized.

Show comment
Hide comment
@hangvar

hangvar Aug 23, 2018

I'm having problems with getting the instrumentation to work despite your clarifications so a repo would be great!

hangvar commented Aug 23, 2018

I'm having problems with getting the instrumentation to work despite your clarifications so a repo would be great!

@paulfalgout

This comment has been minimized.

Show comment
Hide comment
Contributor

paulfalgout commented Aug 23, 2018

@rwwagner90

This comment has been minimized.

Show comment
Hide comment
@rwwagner90

rwwagner90 Aug 26, 2018

@atfzl your coverage code works, but my dist is now full of instrumented code. How do I make sure to not ship the instrumented version?

rwwagner90 commented Aug 26, 2018

@atfzl your coverage code works, but my dist is now full of instrumented code. How do I make sure to not ship the instrumented version?

@paulfalgout

This comment has been minimized.

Show comment
Hide comment
@paulfalgout

paulfalgout Aug 26, 2018

Contributor

In his example with webpack he's setting a flag isDebug to determine whether to add the instrumenter rule or not.

so in the webpack file there'll be something like const isDebug = (process.env.NODE_ENV === 'test');

With babel you'd throw it in the babelrc like:

  "env": {
      "test": {
        "plugins": [ "istanbul" ]
      }
    }

and you'd use something like: https://www.npmjs.com/package/cross-env to set that var in the npm script. cross-env NODE_ENV=test npm run-script coverage

Then don't set the NODE_ENV to test when you're building for dev or prod

Contributor

paulfalgout commented Aug 26, 2018

In his example with webpack he's setting a flag isDebug to determine whether to add the instrumenter rule or not.

so in the webpack file there'll be something like const isDebug = (process.env.NODE_ENV === 'test');

With babel you'd throw it in the babelrc like:

  "env": {
      "test": {
        "plugins": [ "istanbul" ]
      }
    }

and you'd use something like: https://www.npmjs.com/package/cross-env to set that var in the npm script. cross-env NODE_ENV=test npm run-script coverage

Then don't set the NODE_ENV to test when you're building for dev or prod

@rwwagner90

This comment has been minimized.

Show comment
Hide comment
@rwwagner90

rwwagner90 Aug 26, 2018

@paulfalgout that worked, thanks! Now just one more thing I need to figure out. I have a bunch of mocha tests that run first, and I want to merge the coverage results from those into the report we generate from cypress. Any ideas how to do that?

rwwagner90 commented Aug 26, 2018

@paulfalgout that worked, thanks! Now just one more thing I need to figure out. I have a bunch of mocha tests that run first, and I want to merge the coverage results from those into the report we generate from cypress. Any ideas how to do that?

@paulfalgout

This comment has been minimized.

Show comment
Hide comment
@paulfalgout

paulfalgout Aug 27, 2018

Contributor

@rwwagner90 it can likely be done.. you'd need to, after the tests are run in both cases, combine their json output files.
It would be some node similar to what's in the cypress plugin

    const istanbul = require('istanbul-lib-coverage');
    coverageMap = istanbul.createCoverageMap({});

    on('task', {
      'coverage'(coverage) {
        coverageMap.merge(coverage);
        return JSON.stringify(coverageMap);
      }
    });

where you would read in both files and createCoverageMaps out of them and merge them together and write them back out to a single file that nyc uses.

Contributor

paulfalgout commented Aug 27, 2018

@rwwagner90 it can likely be done.. you'd need to, after the tests are run in both cases, combine their json output files.
It would be some node similar to what's in the cypress plugin

    const istanbul = require('istanbul-lib-coverage');
    coverageMap = istanbul.createCoverageMap({});

    on('task', {
      'coverage'(coverage) {
        coverageMap.merge(coverage);
        return JSON.stringify(coverageMap);
      }
    });

where you would read in both files and createCoverageMaps out of them and merge them together and write them back out to a single file that nyc uses.

@rwwagner90

This comment has been minimized.

Show comment
Hide comment
@rwwagner90

rwwagner90 Aug 27, 2018

@paulfalgout I got it working with just nyc. It appears that if you drop coverage output into the coverage folder, nyc automatically merges that report into subsequent runs. It's all working over on https://github.com/shipshapecode/shepherd

rwwagner90 commented Aug 27, 2018

@paulfalgout I got it working with just nyc. It appears that if you drop coverage output into the coverage folder, nyc automatically merges that report into subsequent runs. It's all working over on https://github.com/shipshapecode/shepherd

@hangvar

This comment has been minimized.

Show comment
Hide comment
@hangvar

hangvar Aug 30, 2018

Thanks for all your help @paulfalgout. I managed to get a coverage file generated and nyc producing a report, by ejecting our angular 5 webpack.config.js and configuring instrumentation there. However, I'm dumbfounded by the fact that the coverage file is huge indicating coverage in more or less all our components, which can only lead me to believe that coverage from unit tests in Angular is aggregated with the Cypress coverage. I've struggled to separate these, since I want to be able to see my Cypress coverage only, but with no success. Any advice?

The aggregation is present when retrieving the coverageMap below:

coverageMap = istanbul.createCoverageMap({});

    on('task', {
      'coverage'(coverage) {
        coverageMap.merge(coverage);
        return JSON.stringify(coverageMap);
      }
    });

hangvar commented Aug 30, 2018

Thanks for all your help @paulfalgout. I managed to get a coverage file generated and nyc producing a report, by ejecting our angular 5 webpack.config.js and configuring instrumentation there. However, I'm dumbfounded by the fact that the coverage file is huge indicating coverage in more or less all our components, which can only lead me to believe that coverage from unit tests in Angular is aggregated with the Cypress coverage. I've struggled to separate these, since I want to be able to see my Cypress coverage only, but with no success. Any advice?

The aggregation is present when retrieving the coverageMap below:

coverageMap = istanbul.createCoverageMap({});

    on('task', {
      'coverage'(coverage) {
        coverageMap.merge(coverage);
        return JSON.stringify(coverageMap);
      }
    });
@paulfalgout

This comment has been minimized.

Show comment
Hide comment
@paulfalgout

paulfalgout Aug 30, 2018

Contributor

Well if you can, disable your unit tests or unit coverage and try it again to see if there are any difference.

I suspect what you're seeing is the Cypress coverage cannot tell you what is not covered. It can only tell you what is never touched. This is because the integration of modules and starting your app will touch many lines of code, even before a cypress test is run.

So coverage in integration tests isn't necessarily as useful, but I find it invaluable as it points out likely areas where I'm missing tests and more importantly it helps me prove that I can test a difficult to cover logic branch.

Prior to coverage I had tests where I thought I was covering a particular scenario, but was missing a key step for a obscure use case we wanted to test for.

Contributor

paulfalgout commented Aug 30, 2018

Well if you can, disable your unit tests or unit coverage and try it again to see if there are any difference.

I suspect what you're seeing is the Cypress coverage cannot tell you what is not covered. It can only tell you what is never touched. This is because the integration of modules and starting your app will touch many lines of code, even before a cypress test is run.

So coverage in integration tests isn't necessarily as useful, but I find it invaluable as it points out likely areas where I'm missing tests and more importantly it helps me prove that I can test a difficult to cover logic branch.

Prior to coverage I had tests where I thought I was covering a particular scenario, but was missing a key step for a obscure use case we wanted to test for.

@frattaro

This comment has been minimized.

Show comment
Hide comment
@frattaro

frattaro Sep 14, 2018

Has anyone been able to get the nyc output appended to the cypress run output?

frattaro commented Sep 14, 2018

Has anyone been able to get the nyc output appended to the cypress run output?

@pvdyck

This comment has been minimized.

Show comment
Hide comment
@pvdyck

pvdyck Sep 14, 2018

pvdyck commented Sep 14, 2018

@rwwagner90

This comment has been minimized.

Show comment
Hide comment
@rwwagner90

rwwagner90 Sep 14, 2018

@frattaro yes, we run both mocha tests and cypress tests and merge the coverage at https://github.com/shipshapecode/shepherd

rwwagner90 commented Sep 14, 2018

@frattaro yes, we run both mocha tests and cypress tests and merge the coverage at https://github.com/shipshapecode/shepherd

@najisawas

This comment has been minimized.

Show comment
Hide comment
@najisawas

najisawas Oct 2, 2018

for use with create-react-app, you will need to yarn start with react-app-rewired.

then, in a config-overrides.js file parallel to your src folder, add the following:

const path = require("path");

module.exports = {
  webpack(config) {
    const istanbul = {
      test: /\.(jsx|js)$/,
      include: path.resolve(__dirname, "./src"),
      rules: [
        {
          loader: "istanbul-instrumenter-loader",
          options: {
            esModules: true,
          },
        },
      ],
    };

    config.module.rules.unshift(istanbul);

    return config;
  },
};

this is in addition to installing istanbul-instrumenter-loader, istanbul-lib-coverage, nyc, and adding the above cypress snippets written by @atfzl

najisawas commented Oct 2, 2018

for use with create-react-app, you will need to yarn start with react-app-rewired.

then, in a config-overrides.js file parallel to your src folder, add the following:

const path = require("path");

module.exports = {
  webpack(config) {
    const istanbul = {
      test: /\.(jsx|js)$/,
      include: path.resolve(__dirname, "./src"),
      rules: [
        {
          loader: "istanbul-instrumenter-loader",
          options: {
            esModules: true,
          },
        },
      ],
    };

    config.module.rules.unshift(istanbul);

    return config;
  },
};

this is in addition to installing istanbul-instrumenter-loader, istanbul-lib-coverage, nyc, and adding the above cypress snippets written by @atfzl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment