Error getting failed test output in IE11 without iframe

[kostia]

Expected behavior

Output of a failed test in IE11 gets logged in the console.

Actual behavior

An error is raised:

TypeError: result.suite.join is not a function
    at specFailure (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/karma/lib/reporters/base.js:94:33)
    at onSpecComplete (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/karma/lib/reporters/base.js:80:12)
    at null.<anonymous> (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/karma/lib/events.js:13:22)
    at emitTwo (events.js:100:13)
    at emit (events.js:185:7)
    at onResult (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/karma/lib/browser.js:216:13)
    at Array.forEach (native)
    at onResult (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/karma/lib/browser.js:206:21)
    at Socket.<anonymous> (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/karma/lib/events.js:13:22)
    at emitOne (events.js:95:20)
    at Socket.emit (events.js:182:7)
    at Socket.onevent (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/socket.io/lib/socket.js:335:8)
    at Socket.onpacket (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/socket.io/lib/socket.js:295:12)
    at Client.ondecoded (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/socket.io/lib/client.js:193:14)
    at Decoder.Emitter.emit (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/component-emitter/index.js:134:20)
    at Decoder.add (/Users/kostia/Code/tmp/karma-ie-debug/node_modules/socket.io-parser/index.js:247:12)

Enviroment Details

• "jasmine": "2.4.1",
• "karma": "0.13.22"
• "karma-jasmine": "0.3.6"
• "karma-sauce-launcher": "0.3.0"
• conf.js

module.exports = function(config) {
  config.set({
    singleRun: true,
    frameworks: ['jasmine'],
    files: ['test_spec.js'],
    client: {
      useIframe: false
    },
    browsers: ['ie'],
    customLaunchers: {
      ie: {
        base: 'SauceLabs',
        os: 'Windows 7',
        browserName: 'internet explorer',
        version: '11'
      }
    },
    sauceLabs: {
      startConnect: false,
      tunnelIdentifier: process.env.SAUCELABS_TUNNEL_IDENTIFIER,
      testName: 'debug',
    },
  });
};

Steps to reproduce the behaviour

Simply run a failing spec:

// test_spec.js
describe('when running failing specs on IE11', function() {
  it('raises an error while capturing test output', function() {
    expect(true).toBe(false);
  });
});

The crucial part it the switch client.useIframe. The error is only raise if the iframe is disabled as in my config.

[kostia]

Any progress on this?

[AStoker]

Can you provide more details here?
What is the DOM used for this test? What does the view model look like? Can you recreate the issue with very simple test, outside of your implementation?

[kostia]

@AStoker I've created a minimal setup for reproducing the bug: https://github.com/kostia/karma-runner-karma-issues-2061. Since I'm on OSX, I use Saucelabs for running IE tests. It should however result in the same error in an IE running in a VM. Please let me if you need more info or support on how to use this setup with an IE in a VM.

[AStoker]

I'm not a Karma master here, but trying to help you out where I can :)
For kicks and giggles, does your test work when it is running in say Safari, or Chrome?

[kostia]

I'm not a Karma master here, but trying to help you out where I can :)

Thanks! 😄

does your test work when it is running in say Safari, or Chrome?

The test works when it's running in Safari, Firefox (Linux) and Chrome (Linux). The issues occurs only in IE.

[AStoker]

Could this then be an issue more related to Saucelabs and how it compiles tests for IE?

[kostia]

Unfortunately this problem also appears when the tests are running in the IE running in a VM with no Saucelabs involved. As mentioned, I can create a minimal setup able to reproduce the issue in an IE running in the VirtualBox. Here the setup:

1. First install VirtualBox: https://www.virtualbox.org.
2. Then install a VM with IE11, for example using IEVMs: https://github.com/xdissent/ievms.
3. Start the Windows VM.
4. Check out the recent https://github.com/kostia/karma-runner-karma-issues-2061.
5. From within karma-runner-karma-issues-2061 start the server: npm run serve.
6. Now start IE in the VM and open http://your.host.ip:9876. The test will briefly flicker.
7. Take a look at the the server log on your host. You will notice something similar to:

~/Code/tmp/karma-runner-karma-issues-2061 master npm run serve

> @ serve /Users/kostia/Code/tmp/karma-runner-karma-issues-2061
> karma start karma.serve.conf

31 05 2016 16:32:18.750:INFO [karma]: Karma v0.13.22 server started at http://localhost:9876/
31 05 2016 16:32:23.295:INFO [IE 11.0.0 (Windows 7 0.0.0)]: Connected on socket /#sK94LEKUGmvdRMgxAAAA with id manual-1076
IE 11.0.0 (Windows 7 0.0.0): Executed 0 of 1 SUCCESS (0 secs / 0 secs)
Missing error handler on `socket`.
TypeError: result.suite.join is not a function
    at specFailure (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/karma/lib/reporters/base.js:94:33)
    at onSpecComplete (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/karma/lib/reporters/base.js:80:12)
    at .<anonymous> (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/karma/lib/events.js:13:22)
    at emitTwo (events.js:106:13)
    at emit (events.js:191:7)
    at onResult (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/karma/lib/browser.js:216:13)
    at Socket.<anonymous> (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/karma/lib/events.js:13:22)
    at emitOne (events.js:101:20)
    at Socket.emit (events.js:188:7)
    at Socket.onevent (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/socket.io/lib/socket.js:335:8)
    at Socket.onpacket (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/socket.io/lib/socket.js:295:12)
    at Client.ondecoded (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/socket.io/lib/client.js:193:14)
    at Decoder.Emitter.emit (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/component-emitter/index.js:134:20)
    at Decoder.add (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/socket.io-parser/index.js:247:12)
    at Client.ondata (/Users/kostia/Code/tmp/karma-runner-karma-issues-2061/node_modules/socket.io/lib/client.js:175:18)
    at emitOne (events.js:96:13)
IE 11.0.0 (Windows 7 0.0.0): Executed 1 of 1 (1 FAILED) DISCONNECTED (2.026 secs / 0.001 secs)

NB: If you wish to reproduce the issue on Saucelabs, you now should run npm run sauce.

[AStoker]

Leaning more towards perhaps some issue with Jasmine and it's setup now... Can you put a console.log(result.suite) at the line that is failing inside node_modules/karma/lib/reporters/base.js:94:33. That might help determine what is the result.suite that is trying to call join on. It sounds like it's trying to join the test suites together, but perhaps result.suite is in fact empty, or at least not an array to call join on?...

[kostia]

The output looks like this: { '0': 'when running failing specs on IE11' }. I suppose some layer before should have converted that object to an array, but since it's missing the length property, the conversion failed.

[AStoker]

That does seem to be what could be the issue. Can you do the same thing but run the test in something that works, and then compare the object to what you just posted?

[kostia]

In Chrome (OSX) it is [ 'when running failing specs on IE11' ].

[AStoker]

So now here's when the debugging comes in. Can you maybe put a console.trace() right there and then follow the call stack up to find out where that object is turned into an array?
My bet is that it's using the ES6 Array.from syntax which isn't supported by IE11 (IE will not be supporting ES6, their solution is Edge).

[kostia]

Sorry, I don't get it 😞
Doesn't the error stacktrace already expose the same call stack as in #2061 (comment)?

[AStoker]

Maybe. That was the call stack at the time the error was caught. Sometimes caught errors don't always output all the details of the call stack. But in this case, it's quite possible it's the same. In which case, you can go up that call stack you already have and try and find the location where it's trying to convert the object to an array.

[AStoker]

It would be great if somebody else from the Karma team knew exactly what's going on :P I'm just doing my best to try and guide to a solution.

[kostia]

I could track the problem as far as to https://github.com/karma-runner/karma/blob/master/lib/events.js#L11. In this line Array.prototype.slice should actually build an array from the object of type {0: 'spam', 1: 'eggs', length: 2}, which fails, since the given object is missing the corresponding length property in IE11. In other browsers this property is present.

[AStoker]

I don't believe that length is a typical property in an object. So you get to that line in IE, and the object does not have length as a property. But you get to that line in say Chrome, and length is a property? If that's the case, you might have to dig some more to find out what forms the object, and why isn't it adding the length property to it.

[kostia]

I don't believe that length is a typical property in an object.

Right, it's not typical in an arbitrary object. But an array-like object should have that one, see https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/slice#Array-like_objects.

Since I'm not a Karma expert, I don't know who's building that arrays and how. Unfortunately due to a distributed nature of Karma, it is not obvious, who's actually sending that object 😞

[kostia]

Also I'm wondering, why the hack the issue only appears when running Karma with client: { useIframe: false }. With the iframe enabled there are no problems.

[AStoker]

Yes, an array-like object should have one, but something still makes that array like object, and it would help to identify where that is taking place.

So when running in IE, it works with the useIframe being true, but not false?

[kostia]

Yes, an array-like object should have one, but something still makes that array like object, and it would help to identify where that is taking place.

That's what I'm trying to identify the last 5 hours…

So when running in IE, it works with the useIframe being true, but not false?

Yes. But unfortunately it's not an option for us.

[AStoker]

Those are two big bummers...
Well I finally have some time to pull down your repo and try and running it on my windows vm. I also run on a mac, is there anything special I need to get your sauce labs one running?

[kostia]

I also run on a mac, is there anything special I need to get your sauce labs one running?

You would need to register on Saucelabs, I guess… So just skip it. Running in a VM has the same effect.

[kostia]

Here what I found out so far:

1. https://github.com/karma-runner/karma-jasmine/blob/master/src/adapter.js#L241 provides the results objects with a property log. Regardless on whether the specs are running in an iframe, the object behind log is always a proper array (the node console will render it as […]). This code is executed in the iframe or child window and calls tc.results, which resides in the parent window.
2. When running with the iframe enabled in https://github.com/karma-runner/karma/blob/master/client/karma.js#L128 (which runs in parent window) the object behind the log property is a "normal" object (IE console will render it as […]). However if I disable the iframe (the jasmine code will be executed in a child window, not an iframe), then the object behind log becomes a "strange" array: IE console will render it as {0: "…", length: 1} and JSON.stringify will turn it into something like "{\"0\":\"…\"}" object instead of a properly serialized array ("[\"…\"]").
3. I suppose, that that has something to do with the transition of the corresponding array between the windows and some strange security restrictions in IE.

[kostia]

Here my monkey patch, which converts array like objects to arrays:

  // karma/static/karma.js
  this.result = function (result) {
    if (!startEmitted) {
      socket.emit('start', {total: null})
      startEmitted = true
    }

    if (resultsBufferLimit === 1) {
      // Here comes the fix…
      var fixedResult = {}

      for (var propertyName in result) {
        if (result.hasOwnProperty(propertyName)) {
          var propertyValue = result[propertyName];

          if (Object.prototype.toString.call(propertyValue) === '[object Array]') {
            fixedResult[propertyName] = Array.prototype.slice.call(propertyValue);
          } else {
            fixedResult[propertyName] = propertyValue;
          }
        }
      }

      return socket.emit('result', fixedResult)
    }

    resultsBuffer.push(result)

    if (resultsBuffer.length === resultsBufferLimit) {
      socket.emit('result', resultsBuffer)
      resultsBuffer = []
    }
  }