Add Coverage to e2e tests

[BenjaminSchubert]

Bug Report or Feature Request (mark with an x)

- [ ] bug report -> please search issues before submitting
- [X] feature request

Versions.

N.A.

Repro steps.

N.A.

The log given by the failure.

N.A.

Desired functionality.

I'd like to see end to end tests being able to also have test coverage as for normal tests.

This is useful when doing a project to make sure that nothing is out of the spec.

I'd be glad to help if this feature is accepted, but would need a bit of guidance on getting started.

Thanks a lot !

Mention any other details that might be useful.

[sumitarora]

@BenjaminSchubert AngularCLI uses protractor to run e2e tests, which doesn't support code coverage. angular/protractor#370 Also having e2e do the code coverage won't be that much feasible as you would have to write e2e test for each and every case, branch etc. which can easily be tested using Unit Tests. E2E tests takes more time to run than unit tests so if you have lot of e2e tests covering each and every case it will take lot of time.

[BenjaminSchubert]

@sumitarora Well, there is at least a grunt plugin working for protractor and coverage, which means it is feasible.

I also know that e2e tests are taking much longer to run than unittests. But the aim would be to be able to know roughly if you ran through all your specification or if you have dead code. Moreover, simple tests that do not require unit tests could be covered easily by these. I therefore think that having both coverage is helpful.

[cmelion]

@sumitarora, FWIW we routinely get 100% coverage on our E2E tests for small to medium sized web-apps (when not using the CLI).

    plugins : [{
        path: 'node_modules/protractor-istanbul-plugin',
        outputPath: 'reports/feature/coverage'
    }]

[Stalyon]

Hi @cmelion,
I am interested in this coverage report.
Where did you add these lines? In protractor.conf.js?

[cmelion]

@Stalyon yep (protractor.feature.coverage.conf.js), we have separate conf files for coverage, mostly to avoid dealing with instrumentation obfuscating the code when debugging during normal test runs.

[nrazvan87]

@cmelion I've added this plugin and the settings in protractor.conf.js.
After that, I've ran: ng e2e. No reports are generated, do you have any clues where I should investigate? No extra errors/logs are displayed.

[cmelion]

@nrazvan87 I haven't looked at this in a while, last time I looked the CLI implementation of e2e did not pass the necessary configuration to webpack, so your code is probably not getting instrumented.

You need to have control over both protractor config and your webpack config (see the loader config below):

    if (isE2ETestEnv) {
        // instrument only testing sources with Istanbul, covers js compiled files for now :-/
        config.module.preLoaders.push({
            test: /\.jsx?$/,
            include: path.resolve('src'),
            loader: 'babel-istanbul',
            exclude: [/\.?spec\.js$/, /node_modules/, /test/]
        });
    }

[BenjaminSchubert]

@sumitarora would something around what @cmelion is showing be accepted into angular-cli?

I don't have the knowledge to contribute yet, but if it would be accepted, I can try to make an implementation of it.

[warstick]

@cmelion @BenjaminSchubert
For generating the E2E test cases report in angular4 cli, Where exactly i have to include following piece of code.

if (isE2ETestEnv) {
        // instrument only testing sources with Istanbul, covers js compiled files for now :-/
        config.module.preLoaders.push({
            test: /\.jsx?$/,
            include: path.resolve('src'),
            loader: 'babel-istanbul',
            exclude: [/\.?spec\.js$/, /node_modules/, /test/]
        });
    }

suppose if we can't do this, what is the fallback approach for to generate E2E Coverage report in angular 4 ?

[cmelion]

@warstick I'd have to look at the source for the CLI to see exactly how this could be integrated, probably worthy of a pull-request. I'll try to look into it this weekend.

The alternative is to bypass the CLI altogether which is not ideal.

[jedihacks]

I would also love to see this. Any luck on a configuration to get this supported @warstick @cmelion ?

[artem-v-shamsutdinov]

Same here. Eight now I'm manually instrumenting the bundles and can't even get the TS source mapping to work. Also the coverage information has to be manually pulled from the browser, which is a pain. It would be great if everything was just integrated and just worked.

This is very important for us because we want to tie in what scenarios are covered with what code is covered.

[hojt]

Anyone working on this? @BenjaminSchubert @sumitarora