stryker-karma-runner hangs when Karma can't connect to the browser

[eddpurcell]

Summary

When running stryker with karma, stryker will hang forever during the initial test run if karma fails to connect to the browser. This is most easily tested by running stryker in an environment that proxies all HTTP calls.

This isn't a huge issue, if you turn on log level trace, you can see karma launcher say "ChromeHeadless failed x times (timeout). Giving up.", and then it hangs. The solution to the karma launcher not connecting is to make sure you export NO_PROXY='xxx'. However, stryker-karma-runner should exit on a launcher failure instead of waiting forever.

Stryker config

Taken from the Angular Stryker Example project found here

const getAppFromConfig = require('@angular/cli/utilities/app-utils').getAppFromConfig;
const appConfig = getAppFromConfig();

module.exports = function (config) {
  config.set({
    files: [
      // Exclude files so we can include only particular files later on
      '!./**/*.ts',
      '!./**/*.js',
      `!./${appConfig.root}/**/*.d.ts`,

      // Include source files and flag them for mutation
      { pattern: `./${appConfig.root}/**/*.ts`, included: false, mutated: true },

      // Exclude test files to later include them again without the mutated flag set to true
      `!./${appConfig.root}/**/*.spec.ts`,
      `!./${appConfig.root}/test.ts`,
      { pattern: `./${appConfig.root}/**/*.spec.ts`, included: false, mutated: false },
      { pattern: `./${appConfig.root}/test.ts`, included: false, mutated: false },

      // Ignore environment configuration files
      `!./${appConfig.root}/environments/*.ts`,

      // Other assets (HTML and CSS for the browser)
      `./${appConfig.root}/**/*.html`,
      `./${appConfig.root}/**/*.css`,
    ],
    testRunner: 'karma',
    mutator: 'typescript',
    transpilers: ['webpack'],
    reporter: ['html', 'clear-text', 'progress'],
    testFramework: 'jasmine',
    coverageAnalysis: 'off',
    karmaConfig: {
      frameworks: ['jasmine'],
      browsers: ['ChromeHeadless'],
      plugins: [
        require.resolve('karma-jasmine'),
        require.resolve('karma-chrome-launcher')
      ]
    },
    tsconfigFile: 'tsconfig.json',
    webpack: {
      configFile: 'webpack-stryker.conf.js'
    },
    logLevel: 'info',
    timeoutMs: 10000
  });
};

Stryker environment

Same as the example project.

stryker@0.19.2
stryker-api@0.13.0
stryker-html-reporter@0.12.1
stryker-karma-runner@0.12.2
stryker-typescript@0.9.1
stryker-webpack-transpiler@0.1.4

Your Environment

software	version(s)
node	8.9.1
npm	5.5.1
Ubuntu	16.04

[tommilligan]

+1, ran into this issue while running integration tests locally. angular-project can hang forever if karma fails to setup properly:

22:26:31 (6777) DEBUG ChildProcessProxyWorker Changing current working directory for this process to /home/tom/Documents/stryker/integrationTest/test/angular-project/.stryker-tmp/sandbox3436180
22:26:39 (6777) DEBUG stryker-karma.conf.js Importing config from "/home/tom/Documents/stryker/integrationTest/test/angular-project/.stryker-tmp/sandbox3436180/src/karma.conf.js"
22:26:52 (6760) TRACE ChildProcessProxy 02 10 2018 22:26:52.770:INFO [karma]: Karma v1.7.1 server started at http://0.0.0.0:9336/
                                   
22:26:52 (6760) TRACE ChildProcessProxy 02 10 2018 22:26:52.773:INFO [launcher]: Launching browser ChromeHeadless with unlimited concurrency
                          
22:26:52 (6760) TRACE ChildProcessProxy 02 10 2018 22:26:52.787:INFO [launcher]: Starting browser ChromeHeadless
                  
22:26:52 (6760) TRACE ChildProcessProxy 02 10 2018 22:26:52.789:ERROR [launcher]: No binary for ChromeHeadless browser on your platform.
  Please, set "CHROME_BIN" env variable.

I'd like to treat this as the more generic case of the Sandbox.create function hanging, as we may encounter this with other test runners in future.

edit: this error was thrown while running integration tests for stryker, on a system without chrome installed

[nicojs]

I think this can be a pretty easy fix.

Handle the loadError event in karma:

      StrykerReporter.instance.once('load_error', (type: string, name: string, ...rest: any[]) => {
        rej(new Error());
      });

Should be passed from the StrykerReporter to the KarmaTestRunner.

[nicojs]

I think this is a bigger issue. Not only is load_error effected, but also browser_error. In general, when an error occurs, the promise from a karma run is not resolved, at all.

Time to fix this one.

[rmalveis]

Hey guys. Is this still a problem or it is already solved on #1832 ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.