Karma as end to end test runner

[arqex]

Hey guys!

I love the way karma works, and I am decided to use it as my end to end test runner and get rid of selenium, its asynchronous tests and the way it got stuck without any clear reason.

I already have some code to show, let me explain some context:

The tests are running inside of my app, let's say in mydomain.com.

Karma will be running in my computer or any CI environment, like Bamboo, Travis or Codeship, so it will be able at localhost:9876.

This domain mismatch is an issue that can be avoided not using iframes for the testing, and relaxing karma's server Access-Control-Allow-Origin, this is a piece of the karma.conf.js file:

{
    client: {
      useIframe: false,
      runInParent: true
    },
    customHeaders: [{
      match: '.*',
      name: 'Access-Control-Allow-Origin',
      value: '*'
    }]
}

And now I need to load karma's environment in my app, and to do so I have created a small script. It works:

var karmaURL = 'http://localhost:9876';

// A helper to load scripts from karma's server
var load = function( path, clbk ){
  var script = document.createElement('script');
  script.src = karmaURL + path;
  script.onload = clbk;
  document.body.appendChild( script );
}

load('/socket.io/socket.io.js', () => {
load('/karma.js', () => {

// HACK: Updates socket.io uri to point to karma server
karma.socket.io.uri = karmaURL;

load('/context.js', () => {
  // HACK: Get the configuration
  fetch('http://localhost:9876/debug.html')
    .then( res => res.text() )
    .then( text => {

      // Load configuration and script list
      var config = text.match(/Configure.*?\n([^<]*)<\/script>/)[1];
      eval( config );


      /* Karma should load the files for me but it didn't work
      // The list of scripts will be at `window.__karma__.files`
      var files = Object.keys( window.__karma__.files )
        .map( path => karmaURL + path )
      ;

      window.__karma__.scriptUrls = files;
      window.__karma__.loaded();
      */

      // HACK: Since I can't make karma loading the file, load them manually
      window.__karma__.scriptUrls = [];

      // The list of scripts will be at `window.__karma__.files`
      var files = Object.keys( window.__karma__.files );

      function loadN( i ){
        i < files.length ?
          load( files[i], () => loadN(i+1) ) :
          window.__karma__.loaded()
        ;
      }
      loadN(0);
    })
  ;
})
})
});

If I add this script to my page and start karma server, the tests are run when I open my website, and it works! I can start clicking here and there and check the results of my tests interactions, but it's so hacky.

There are so many hacky parts, that maybe you can help me to make it better:

• When the tests are launched, it always launch http://localhost:9876 and I don't need it anymore. It make the tests fail because they are not in the proper context. It would be great to be able of configure a way of start the browser with the URL of my site: mydomain.com/?tests=1.
• Socket.io is always been initialized pointing at the server that is running the tests, in this case mydomain.com, it should always point to Karma's server. You see I fix it adding karma.socket.io.uri = karmaURL after I load Karma, but Karma still tries to connect to the ws in mydomain.com once.
• The script uses a regular expression to extract the configuration from debug.html. And then uses eval. Is there anything hackier than using eval? Should be there something simpler and safer to get the config?
• I detected that I fill window.__karma__.scriptUrls with the scripts, karma tries to load them automatically, but it was returning some error, so the script loads them by itself.

Any help on how to improve the code, or doing it in a more standard way is welcome.

[EzraBrooks]

Currently, Karma isn't really designed for end-to-end testing. However, it is in the backlog to integrate web driver support. For more information, take a look at #413.

[ghost]

@arqex
What I would do is adding an iframe to the karma client and load the actual page (dev.xy.com) there. That page should contain the CORS header and allow the karma client (localhost:5678) to manipulate it. From that point your tests should wait until the iframe loads and after that you can follow the links or submit the forms with your tests. I think it is doable, I'll try to implement it this week.

What you were doing is including karma into your app, which is only good for single page applications (ajax, socket.io). By a regular web application your tests will break by sending a form or following a link. I am not sure whether testing the production server with mydomain.com is a good idea. Maybe you can measure loading speeds with it, but for feature testing it is a lot better to run the app locally for example in a docker container.

update:
It works with iframes and you don't need CORS if you disable browser security with flags.

[ghost]

Meanwhile I started an end to end testing library which supports Karma: https://github.com/inf3rno/e2e and uses client side scripting. I ended up with window.open() and disabled web security, because unlike iframes, that does not interfere with the security measures of the tested site, and I can serve the tests on a different origin and I can access the tested page in a loading readyState before any script runs, so I can override the console to report to the parent window for example. This is the only working solution as far as I know. Everything else has major issues. The only problem with this solution that it entirely depends on what the Chromium developers do with the --disable-web-security flag, so I had to join to their google group and take part in that discussion.