Configurable whether process should exit with non-zero exit code when tests failed?

[davidparsson]

We run tests on Jenkins using Karma and Grunt. Karma produces JUnit reports using the karma-junit-reporter, and we've configured Jenkins to set the build as unstable (yellow) if test failures are found in these report files.

The problem is that whenever a test fails, Karma (and Grunt) exits with a non-zero exit code and this makes the build failed (red).

I think the process still should fail in all those cases where the tests haven't been completely executed, for example when a browser has been disconnected.

Is there any way to configure this today? If not, I'll gladly provide a pull request.

[mightyiam]

I'm quite sure you can do this if you run Karma through its API, instead of through karma-cli. See API.

[davidparsson]

I don't think so. I found no way to distinguish if there were failed assertions in the tests or if the tests failed to execute, for example if a browser disconnected. Do you?

The API only callbacks with the exit code, and the exit code can be 1 both if one test fails and if a browser disconnects.

[mightyiam]

@davidparsson It seems that you're right. It may be that what you can use a reporter for this purpose, while also running through the API.

[dignifiedquire]

I think the way to do this to expand the api support to give more insight to the exit reasons of Karma and then add an option for this to grunt-karma. Happy to accept a pr for this

[larryhe]

Hi @davidparsson, how did you fix your problem? I had the exactly same issue that running karma tests will have random DISCONNECTED issue (for example, If I run the same tests 10 times on my local, I probably will see that DISCONNECTED issue 2~3 times). Even worse, karma exit code is 1 on DISCONNECTED issue. In this case,I want the exit code 0 because technically there is no any unit tests failure. Or is there any way to distinguish unit test failure and DISCONNECTED? Any suggestion is highly appreciated. Thanks for your time.

[davidparsson]

@larryhe, this is still an issue for me. For your disconnect issues, take a look at browserDisconnectTimeout, browserDisconnectTolerance and browserNoActivityTimeout in the docs.

[davidparsson]

@dignifiedquire, I think the use-case described in the feature request could be desirable when running karma from the CLI as well. Would you consider allowing this to be a configured using a karma.conf.js/command-line option?

[davidparsson]

Sadly my situation has changed, so I'm not likely going to submit a PR for this in the near future.

[ianwremmel]

I'm looking for something similar. I'd like to treat test failures as successes when generated structured output. Since karma seems to always exit with code 1, whether as a result of test failures or e.g. failure to get a browser, I can't use my CI system to determine the difference.

[costincaraivan]

I'm kind of surprised that this is the case. Is no one using Karma with tools like Jenkins? :(

[kintax]

I do. We are using Karma w/ ChromeHeadless, and trying to figure out how to run tests in Jenkins and have failing tests NOT fail the build. Ultimately we want them to fail the build but we're doing heavy dev right now so we want the UT fails to be more like warnings.

[andreaspsson]

I also run tests using Jenkins and have the same problem. Would be really neat with a proper solution for this problem

[dougludlow]

Just to help any other googlers, the failOnFailingTestSuite option was added. Looks like maybe --no-fail-failing-test-suite could work from the command line?

[jslegers]

I'm having sort-of the opposite problem here.

I'm starting my Karma like this (in Java) :

ArrayList<String> karmaTestCommand = new ArrayList<String>();
boolean isWindows = System.getProperty("os.name").startsWith("Windows");
if (isWindows) {
  karmaTestCommand.add("cmd"); //on Windows, NPM is not an executable, but a .cmd file
  karmaTestCommand.add("/c");
}
karmaTestCommand.add("npm");
karmaTestCommand.add("run");
karmaTestCommand.add("test");

ProcessBuilder karmaProcessBuilder = new ProcessBuilder(karmaTestCommand);
karmaProcessBuilder.inheritIO();
Map<String, String> karmaProcessEnv = karmaProcessBuilder.environment();
karmaProcessEnv.put("RIA_TESTSERVER_PORT", Integer.toString(testserverPort));
karmaProcessEnv.put("KARMA_SERVER_PORT", Integer.toString(karmaServerPort));
Process karmaProcess = karmaProcessBuilder.start();

boolean completed = new ProcessWatchDog(karmaProcess).waitFor(20, TimeUnit.MINUTES);
if (!completed) {
  System.err.println("Tests timed out after 20 minutes!");
  RIATestReport.writeTestTimeoutReport("target/junit_reports/test-results.xml", RunKarmaTests.class.getName());
}

This starts the following script :

"scripts": {
  "test": "karma start common/test/karma/karma.conf.js --singleRun=true || (exit 0)"
}

Due to some issue with Firefox on one of our Jenkins slaves, Karma behaves like this :

[java] 02 04 2019 14:06:25.834:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
[java] 02 04 2019 14:06:25.891:DEBUG [launcher]: Process Firefox exited with code 0
[java] 02 04 2019 14:06:25.891:DEBUG [temp-dir]: Cleaning temp dir C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:06:25.897:INFO [launcher]: Trying to start Firefox again (1/2).
[java] 02 04 2019 14:06:25.897:DEBUG [launcher]: Restarting Firefox
[java] 02 04 2019 14:06:25.897:DEBUG [temp-dir]: Creating temp dir at C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:06:25.898:DEBUG [launcher]: C:\Program Files\Mozilla Firefox\firefox.exe http://localhost:51005/?id=95416947 -profile C:\Users\release\AppData\Local\Temp\karma-95416947 -no-remote
[java] 02 04 2019 14:07:25.898:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
[java] 02 04 2019 14:07:25.961:DEBUG [launcher]: Process Firefox exited with code 0
[java] 02 04 2019 14:07:25.961:DEBUG [temp-dir]: Cleaning temp dir C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:07:25.967:INFO [launcher]: Trying to start Firefox again (2/2).
[java] 02 04 2019 14:07:25.967:DEBUG [launcher]: Restarting Firefox
[java] 02 04 2019 14:07:25.967:DEBUG [temp-dir]: Creating temp dir at C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:07:25.968:DEBUG [launcher]: C:\Program Files\Mozilla Firefox\firefox.exe http://localhost:51005/?id=95416947 -profile C:\Users\release\AppData\Local\Temp\karma-95416947 -no-remote
[java] 02 04 2019 14:08:25.968:WARN [launcher]: Firefox have not captured in 60000 ms, killing.
[java] 02 04 2019 14:08:26.038:DEBUG [launcher]: Process Firefox exited with code 0
[java] 02 04 2019 14:08:26.038:DEBUG [temp-dir]: Cleaning temp dir C:\Users\release\AppData\Local\Temp\karma-95416947
[java] 02 04 2019 14:08:26.045:ERROR [launcher]: Firefox failed 2 times (timeout). Giving up.

This causes Karma to become unresponsive and will result in the process reaching a timeout after 20 minutes.

Is there any was I can get this process to end immediately, so I don't block my Jenkins slave for nothing?