Angular Cli: 6.0.1 code coverage thresholds not failing ng test task

[banjankri]

After upgrade to Angular Cli 6.x the code coverage thresholds enforced by the karma-coverage-istanbul-reporter do not make the testing command fail with emitWarning set to false as they used to when using Angular Cli 6.0.1.

Versions

Angular CLI: 6.0.1
Node: 8.9.1
OS: win32 x64
Angular: 6.0.1
... animations, cli, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic
... platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.1
@angular-devkit/build-angular     0.6.1
@angular-devkit/build-optimizer   0.6.1
@angular-devkit/core              0.6.1
@angular-devkit/schematics        0.6.1
@ngtools/webpack                  6.0.1
@schematics/angular               0.6.1
@schematics/update                0.6.1
rxjs                              6.1.0
typescript                        2.7.2
webpack                           4.6.0

Repro steps

1. Create a repo using Angular Cli 1.7.x
2. Enforce thresholds
3. See the test command failing
4. Upgrade Angular Cli and migrate

Observed behavior

The error is emitted by the reporter, however seems ignored by the angular client.

Desired behavior

The test command should be failing when thresholds not met as it used to.

Mention any other details that might be useful (optional)

Tried with both karma@1.7.x and 2.x, reporter version 0.x and 1.x. Seems that stops working as soon as the upgrade of Angular Client is done. Same repo with Angular Client 1.7 behaves correctly.

[mboughaba]

We are running into the same issue. Status code of the ng test --code-coverage will always be 0.
We are heavily relying on this feature in our local and CI environment.

[mboughaba]

@filipesilva can you please let us know if you think this is an issue with cli? if so is there a workaround?
any idea where to start the investigation?

[DanKotowski]

Our team is also experiencing the same issue.

[elleneub]

My team also has this issue and it's fairly integral to our workflow. Hope this can get prioritized, thanks!

[bennyadam]

Same issue here, currently preventing our regular CI workflow from running as expected

[abhishekgoenka]

I am also facing the same issue with cli 1.7.4

[banjankri]

any update on the issue please? it is preventing us from upgrading to version 6.x.

[bmarti44]

After upgrading from cli version 1.6.6 to 6.0.8, it looks like the exit status of the process is always 0, even if the code coverage thresholds are not met. An example from our docker build ->

Step 29/32 : RUN npm test
 ---> Running in fe2f19a2f026

> nba-angular@6.0.0 test /nba-angular
> ng test game --browsers HeadlessChrome --code-coverage=true --watch=false && mkdir dist/coverage && mv coverage/* dist/coverage/

 10% building modules 1/1 modules 0 active10 07 2018 23:08:09.424:WARN [watcher]: Pattern "/nba-angular/styles.css" does not match any file.
(node:17) DeprecationWarning: Tapable.plugin is deprecated. Use new API on `.hooks` instead
 10% building modules 3/4 modules 1 active /nba-angular/src sync /\.spec\.ts$/10 07 2018 23:08:18.400:INFO [karma]: Karma v2.0.4 server started at http://0.0.0.0:9876/
10 07 2018 23:08:18.400:INFO [launcher]: Launching browser HeadlessChrome with unlimited concurrency
10 07 2018 23:08:18.406:INFO [launcher]: Starting browser ChromeHeadless
10 07 2018 23:08:31.078:WARN [watcher]: Pattern "/nba-angular/styles.css" does not match any file.
10 07 2018 23:08:31.396:INFO [HeadlessChrome 0.0.0 (Linux 0.0.0)]: Connected on socket hqPyzaLHxba-nxi3AAAA with id 29738065
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 4 of 84 SUCCESS (0 secs / 0.417 secs)
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 21 of 84 SUCCESS (0 secs / 0.868 secs)
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 22 of 84 SUCCESS (0 secs / 0.925 secs)
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 24 of 84 SUCCESS (0 secs / 1.038 secs)
HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 84 of 84 SUCCESS (3.534 secs / 3.039 secs)
TOTAL: 84 SUCCESS
TOTAL: 84 SUCCESS
TOTAL: 84 SUCCESS
TOTAL: 84 SUCCESS
10 07 2018 23:08:39.528:ERROR [reporter.coverage-istanbul]: Coverage for statements (27.13%) does not meet global threshold (31%)
10 07 2018 23:08:39.529:ERROR [reporter.coverage-istanbul]: Coverage for lines (26.9%) does not meet global threshold (31%)
Removing intermediate container fe2f19a2f026
 ---> 7a0b121e05a5
Step 30/32 : RUN node build-apps.js
 ---> Running in 9f9af07796d0
Building projects game,search,video

Angular CLI information

Angular CLI: 6.0.8
Node: 9.6.1
OS: darwin x64
Angular: 6.0.7
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.8
@angular-devkit/build-angular     0.6.8
@angular-devkit/build-optimizer   0.6.8
@angular-devkit/core              0.6.8
@angular-devkit/schematics        0.6.8
@angular/cdk                      6.3.2
@angular/cli                      6.0.8
@ngtools/webpack                  6.0.8
@schematics/angular               0.6.8
@schematics/update                0.6.8
rxjs                              6.2.1
typescript                        2.7.2
webpack                           4.8.3

For the interim, I'm going to attempt to write this to a file, and fail our build if I can scrape the error message. But, it's definitely not ideal.

[cheshirsky]

Same issue with the exit code being 0. The output:

⇒  ng test --browsers Chrome --code-coverage=true --watch=false; echo $?
11 07 2018 14:34:44.604:INFO [karma]: Karma v2.0.4 server started at http://0.0.0.0:9876/
11 07 2018 14:34:44.606:INFO [launcher]: Launching browser Chrome with unlimited concurrency
11 07 2018 14:34:44.611:INFO [launcher]: Starting browser Chrome
11 07 2018 14:34:51.569:INFO [Chrome 67.0.3396 (Mac OS X 10.12.6)]: Connected on socket hvsiXwNm6-QYSfFIAAAA with id 90762867
Chrome 67.0.3396 (Mac OS X 10.12.6): Executed 26 of 28 (skipped 2) SUCCESS (0.4 secs / 0.332 secs)
TOTAL: 26 SUCCESS

=============================== Coverage summary ===============================
Statements   : 56.86% ( 348/612 )
Branches     : 12.67% ( 19/150 )
Functions    : 34.22% ( 64/187 )
Lines        : 56.89% ( 293/515 )
================================================================================
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for statements (56.86%) does not meet global threshold (100%)
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for lines (56.89%) does not meet global threshold (100%)
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for branches (12.67%) does not meet global threshold (100%)
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for functions (34.22%) does not meet global threshold (100%)
0

And the ng --version output:

Angular CLI: 6.0.8
Node: 8.9.4
OS: darwin x64
Angular: 6.0.3
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.8
@angular-devkit/build-angular     0.6.8
@angular-devkit/build-optimizer   0.6.8
@angular-devkit/core              0.0.29
@angular-devkit/schematics        0.0.52
@angular/cli                      6.0.8
@ngtools/json-schema              1.1.0
@ngtools/webpack                  6.0.8
@schematics/angular               0.6.8
@schematics/update                0.6.8
ng-packagr                        2.2.0
rxjs                              6.1.0
typescript                        2.7.2
webpack                           4.8.3

[codeNoobie]

@hansl Is this fixed? We tried the latest rc version but didn't work for us.

[alexw10]

@codeNoobie you need to install latest version of the:
"@angular-devkit/build-angular": "~0.7.1",

[sharathgg5]

"@angular-devkit/build-angular": "~0.7.1",
did not fix the issue

[codeNoobie]

@alexw10 We have tried that but it doesn't seem to work. Could you please check again if this issue has been fixed ?

[wosi00]

Hi there,

The same issue is still persist for us as well, using
"@angular-devkit/build-angular": "~0.7.5",
Also tried 0.8.0 RC - same result.

Can you please check?

[croso-df]

After trying everything, I was able to make it work by upgrading @angular-devkit/build-angular to 0.7.5 as per @alexw10 suggestion. My config looks like this:

coverageIstanbulReporter: {
  verbose: true,
  dir: require('path').join(__dirname, 'coverage'),
  reports: ['html', 'lcovonly'],
  fixWebpackSourcePaths: true,
  thresholds: {
    emitWarning: false, // <- this is important to make karma fail
    global: {
      statements: 85,
      lines: 85,
      branches: 85,
      functions: 85
    }
  }
}

Also, don't forget to set singleRun to false when running the tests with --code-coverage.

singleRun: config.angularCli && config.angularCli.codeCoverage

[patrickmichalina]

This is not fixed.

[SlashmanX]

Found a fix for this:

Remove coverage-istanbul from your reporters array in karma.conf and it will use the correct exit code.

It seems that coverage-istanbul is registered automatically when codeCoverage is set to true in your configuration and when it was being called twice it was causing issues.

[wosi00]

Thank you! I've tried it and it did the trick for us! Eóin Martin <notifications@github.com> ezt írta (időpont: 2018. nov. 6., K, 16:10):

…

Found a fix for this: Remove coverage-istanbul from your reporters array in karma.conf and it will use the correct exit code. It seems that coverage-istanbul is registered automatically when codeCoverage is set to true in your configuration and when it was being called twice it was causing issues. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10940 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ao8vKcmC0cD9XaB6Hlg0fOanvZbh8uyxks5usaZTgaJpZM4UGznr> .

[alan-agius4]

Hi, when using karma-parallel you need to aggregate the code coverage report from different browsers.

This can be done by setting the aggregatedReporterTest.

    parallelOptions: {
      executors: 4, // Defaults to cpu-count - 1
      shardStrategy: 'description-length',
      aggregatedReporterTest: '/coverage|istanbul/'
    },

It is important to note, that the RegExp needs to be passed as a string.

[alan-agius4]

Apart from that I am unable to replicate the issue that when the code coverage is below the threshold it is not existing with a non zero error code.

npm run test -- --code-coverage --watch=false

> ng test "--code-coverage" "--watch=false"

 30% building 16/16 modules 0 active20 06 2019 08:52:16.417:INFO [karma-server]: Karma v4.1.0 server started at http://0.0.0.0:9876/
20 06 2019 08:52:16.425:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
20 06 2019 08:52:16.435:INFO [launcher]: Starting browser ChromeHeadless                                                                                         20 06 2019 08:52:22.262:INFO [HeadlessChrome 75.0.3770 (Windows 10.0.0)]: Connected on socket PibQn-_TY8h99UXSAAAA with id 61228163
HeadlessChrome 75.0.3770 (Windows 10.0.0): Executed 5 of 5 SUCCESS (0.32 secs / 0.249 secs)
TOTAL: 5 SUCCESS

=============================== Coverage summary ===============================
Statements   : 100% ( 10/10 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 3/3 )
Lines        : 100% ( 7/7 )
================================================================================
20 06 2019 08:52:25.723:ERROR [reporter.coverage-istanbul]: Coverage for statements (100%) does not meet global threshold (200%)
TOTAL: 5 SUCCESS
TOTAL: 5 SUCCESS
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! eight-pwa@0.0.0 test: `ng test "--code-coverage" "--watch=false"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the eight-pwa@0.0.0 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\alag\AppData\Roaming\npm-cache\_logs\2019-06-20T06_52_26_003Z-debug.log

What is slightly confusing is that the coverage error is appearing below the TOTAL: 5 SUCCESS success message

[alan-agius4]

Hi I'll leave this this a couple of days open to see if anyone provide a reproduction were when using --code-coverage together with emitWarning: false the process is not exiting with a non zero error code. (Providing the karma.conf.js should be enough)

At this point, I am unable to replicate any of the issues reported. Thanks.

[alan-agius4]

Closing, as no further details provided.

If the issue persists kindly open a new issue with a reproduction.

Thanks.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}