Tests are actually running in all browser instances

[ghost]

• **I'm submitting a ... **

  • [ X] bug report
  • feature request
  • support request => Please do not submit support request here, see note at the top of this template.

• Do you want to request a feature or report a bug?
  Bug

• What is the current behavior?
  I'm getting all tests running on every browser instance.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via

  • set karma config like:

    {
        plugins: ['karma-chrome-launcher', 'karma-coverage', 'karma-junit-reporter', 'karma-mocha', 'karma-parallel'
        ],
        basePath: './',
        frameworks: ['parallel', 'mocha'],
        parallelOptions: {
            shardStrategy: 'round-robin',
            aggregatedReporterTest: /coverage|istanbul|junit/i
        },
        files: [
// stripped for brevity
            '../test/**/*.spec.js'
        ],
        preprocessors: {
            'scripts/**/*.js': 'coverage'
        },
        proxies: {
            '/apps': 'http://localhost:9000/apps',
            '/images': 'http://localhost:8080/images',
            '/resources': 'http://localhost:8080/resources',
            '/foo': 'http://localhost:8080/foo'
        },
        exclude: [],
        reporters: ['progress', 'coverage', 'junit'],
        junitReporter: {
            outputFile: 'test-results.xml',
            suite: ''
        },
        coverageReporter: {
            type: 'cobertura',
            dir: 'coverage/'
        },
        port: 9876,
        runnerPort: 9100,
        colors: true,
        logLevel: config.LOG_INFO,
        autoWatch: true,
        browsers: ['HeadlessChrome'],
        customLaunchers: {
            'HeadlessChrome': {
                base: 'ChromeHeadless',
                flags: [
                    '--no-sandbox'
                ]
            }
        },
        browserNoActivityTimeout: 120000,
        captureTimeout: 120000,
        singleRun: false
    }

• Setup an desribe.only()
• Within the only, create several it()s that throws "whatever1" ... to make them throw when the tests run
• run the tests

When you look at the junit report, it has 3 (cpus-1) instances in it:

  <testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
    <failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
  </testcase>
  <testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0.001" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
    <failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
  </testcase>
  <testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
    <failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
  </testcase>

Also, the console shows 3:

            HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
     [exec]     Error: the string "crap1" was thrown, throw an Error :)
            HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
     [exec]     Error: the string "crap1" was thrown, throw an Error :)
            HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
     [exec]     Error: the string "crap1" was thrown, throw an Error :)

• What is the expected behavior?
  I'm actually not sure how this would look when it works, but I would assume that a test would get skipped in "other" browsers, and, and I don't see any tests getting skipped.

• What is the motivation / use case for changing the behavior?
  The current behavior makes the plugin worthless. (I hope there is something weird about my settings or versions being used, as I can't imagine this is how it works for everyone else?)

• Please tell us about your environment:

  • version: 0.2.4
  • Browser: ChromeHeadless 65.0.3325
  • Language: all
  • package.json contains these dependencies:
    "chai": "1.10.0",
    "gulp": "3.9.1",
    "gulp-util": "3.0.8",
    "istanbul": "0.4.5",
    "karma": "2.0.0",
    "karma-chai": "0.1.0",
    "karma-chrome-launcher": "2.2.0",
    "karma-coverage": "1.1.1",
    "karma-junit-reporter": "1.2.0",
    "karma-mocha": "1.3.0",
    "karma-parallel": "0.2.4",
    "mocha": "2.3.2",
    "sinon": "1.10.2",
    "sinon-chai": "3.0.0"

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

[ghost]

For example, when running 4 executors, with 3297 tests, the progress reporter looks like this:

     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1225 of 3297 (3 FAILED) (0 secs / 19.645 secs)
     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1222 of 3297 (3 FAILED) (0 secs / 17.717 secs)
     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1224 of 3297 (3 FAILED) (0 secs / 19.068 secs)
     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1225 of 3297 (3 FAILED) (0 secs / 17.414 secs)

[joeljeske]

Interesting. Your expectations of this plugin is correct. I am not getting that behavior. Can you create a simple project that reproduces this?

Also, you might try it without the '--no-sandbox' custom browser flag. This plugin has to distinguish between each browser instance and it does so by using a cookie on initial load. If they are not sandboxed, it might cause a race condition in which 2 or more different browser instances, may operate on the same set of test files. (I have tried turning on --no-sandbox in my test project but am not getting incorrect results but it may still be worth trying)

[joeljeske]

@Matt812 Did you ever try to remove the --no-sandbox flag? I'm just curious if that is an issue that should be mentioned in the readme.

[ghost]

@joeljeske , honestly, I can't remember now. We have a very large test suite, and I found that the performance of the karma-parallel method was too slow due to the whole suite being loaded in every browser, so ended up going with the karma-sharding package, which scaled better, so have not looked at this since. Sorry! -- Matt

[AustinMatherne]

I ran into the same issue. However, I was using PhantomJS and switching to ChromeHeadless resolved it for me. Obviously, mine had a different cause than the OP, but I figured I should post this here in case someone else comes along with the same problem.

[joeljeske]

Closing due to inactivity.

[sky007]

Have the same problem with karma-parallel and mocha. They seems to run the all the test in all of the browsers, I debuged it and the ctx.it and so on, doesn't override it of the mocha ones.

This is the result of karma-parallel with mocha:

12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:35.597 �[1A�[2K�[1A�[2K�[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.139 secs / 2.898 secs)
12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (0 secs / 2.799 secs)
12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:35.799 �[1A�[2K�[1A�[2K�[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.139 secs / 2.898 secs)
12:56:35.799 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.747 secs / 2.799 secs)
12:56:35.799 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:36.101 TOTAL: 2133 SUCCESS
12:56:36.101
12:56:36.101 Finished in 5 mins 52.31 secs / 8.374 secs @ 12:56:36 GMT+0200 (CEST)
12:56:36.101
12:56:36.101 SUMMARY:
12:56:36.101 ✔ 2133 tests completed
12:56:36.101 ℹ 54 tests skipped
12:56:36.101 TOTAL: 2133 SUCCESS

and this is what I get from running without karma-parallel:

16:27:07.184 �[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (ski SUCCESS (1 min 22.59 secs / 2.849 secs)
16:27:07.185 TOTAL: 711 SUCCESS
16:27:07.185 TOTAL: 711 SUCCESS
16:27:08.89
16:27:08.89 Finished in 1 min 22.59 secs / 2.849 secs @ 16:27:07 GMT+0200 (CEST)
16:27:08.89
16:27:08.89 SUMMARY:
16:27:08.89 ✔ 711 tests completed
16:27:08.89 ℹ 18 tests skipped

[premranderia]

@sky007 I am facing the same issue ? Did you find any resolution for the above ?
@joeljeske Can we re-open this issue ? I tried without sandbox flag and still the issue