JavaScript Tests

Tyler Hansen edited this page Aug 1, 2015 · 3 revisions

JavaScript tests in Windmill consist of a directory tree full of .js files containing your tests. On the server-side, Windmill just recursively parses the directory tree and serves up the files inside. When you run your tests, Windmill evals the contents of your test .js files in the window scope of the application being tested. This just means that if you have `someNamespace` in your app, you can access it directly in your JavaScript tests without doing anything special.

The JavaScript test engine can call the same set of UI commands as Python tests, and also allows you to use all the same asserts available in both of the major JSUnit testing frameworks (see “Asserts,” below).

Test Setup and Parsing

Windmill will search recursively through the tested applications’s `window` object and register any objects that have a name beginning with “test_” (‘test’ plus underscore), “setup”, or “teardown.”

Test Phases

The “setup”/“test_”/“teardown” naming determines what phase the tests belong to: within the same level, tests named “setup” are run first, followed by any named “test_”, followed by any named “teardown.” This allows you to create a complex hierarchy of tests, and make sure that whatever setup is needed for a specific test happens exactly the same way every time.

When you’re running tests, there’s also a way to filter them down to just the ones in a particular phase (see “Test-Writing Workflow,” below). This saves a lot of time when you’re developing tests, by cutting out redundant steps, and can help in debugging by allowing you to home in on a single phase (e.g., “Why won’t the stupid teardown work?”).

Organizing Your Tests

The `require` command can be used in your !JavaScript tests to share code between modules, or to resolve order-of-loading issues. `windmill.jsTest.require` forces the framework to load and eval a file immediately. You call it like this:

windmill.jsTest.require('shared/test_login.js');

The path for the require is always relative to the location of the top-level directory of !JavaScript tests you’re running — NOT to the file where you’re doing the `require`. (This is a tradeoff — it’s not very flexible, but we don’t have to think about how to map !JavaScript namespaces to directory and file names.)

The `require` method works recursively — you can `require` files that `require` other files. The framework will only eval the code in a particular file once during a test run, no matter how many times you `require` it.

The initialize.js file

Windmill JavaScript tests gives you one other important tool for the order-of-loading problem: before loading, evaling, and parsing test files, it looks for a file named initialize.js in the root directory of your tests, and runs the code in it first, before doing anything with any other files in your test directory.

This can be useful for setting up shortcuts to use in your tests, or creating namespaces for holding your test objects temporarily until you’re ready to hang them on your test namespace hierarchy.

An initialize.js file might have stuff in it like this:

var BASE_PATH = '/some_path'
var ACCOUNT_PREFS = myApp.base.getPrefs();
var USERNAME = fleegix.cookie.get('username');
var LOGOUT_REDIRECT = 'http://subdomain.mydomain.org/thanks';

var fooNamespace = {};
var barNamespace = {};

Basically, just remember that any code in initialize.js will run before anything else in your test code.

Running Your Tests in a Specific Order

If you just drop a bunch of .js files in a test directory, and run your tests, there is NO GUARANTEE OF THE ORDER in which your files will be evaled and parsed.

The !JavaScript test engine lets you specify the order you want your tests to run in, by calling the `register` command in your initialize.js file, and passing it an array of the test namespaces you want to run, in your desired order, like so:


// Use this to set the order you want your tests to run
var registeredTests = [
  'test_jsonDom',
  'test_waitForXHR',
  'test_formBasics',
  'test_jumBasics',
  'test_scope',
  'test_timer'
];

// Register top-level test namespaces in the order
// we want to run them
windmill.jsTest.register(registeredTests);

The tests inside the namespace will run in the order you define them — except for any setup or teardown, which run first and last, respectively.

Test Formats

JavaScript tests come in in two flavors:

  • Executable functions
  • Windmill controller API commands objects

You can write these tests to run individually, or you can put into an array and run them in aggregate. (You can also organize tests into namespaces — see “Namespacing Tests,” below.)

Single functions

Single-function tests are just !JavaScript functions executed in the window scope of the application you’re testing.

Here are a couple of examples:

function test_fooThing () {
  var foo = 'asdf';
  jum.assertEquals(foo, 'asdf');
  var bar = 'qwer';
  jum.assertNotEquals(bar, 'asdf');
}

var test_getData = function () {
  // Callback function -- sets temp flag to 'returned'
  var success = function (s) {
    var data = eval('('+s+')');
    myTestInitData.tempData = data;
  };
  var url = windmillMain.shared.util.getCurrentDir() + 'data.json';
  fleegix.xhr.get(success, url);
};

Controller API Command Objects

All the actions in the Controller API are available from the JavaScript test framework.

The big difference in the JavaScript tests from Python tests is that there is no server round-trip for each action — the JavaScript test engine simply passes the actions directly to the controller object, so test execution is generally faster. However this also means that reporting on success/failure doesn’t occur until the entire test run completes.

Asserts

var test_albumNodeExists = { 
  method: "asserts.assertNode", 
  params: {
    id: 'album_9'
  } 
};
var test_albumNodeText =  {
  method: "asserts.assertText",
  params: {
    id: 'album_9',
    validator: 'Power Windows'
  }
};

Waits

test_hasNavigated = {
  method: "waits.forElement",
  params: { 
    id: "formPageHeader"
  } 
};

Warning: keep in mind that note that code containing these kinds of UI control objects is eval’d inline when the code for that .js test file is loaded. You can’t reference objects in your command objects that don’t exist on the page when the test code is eval’d.

Calling actions directly from JavaScript

Note that the UI command-object methods are also available as JavaScript method calls on the `windmill.jsTest.actions` object. This can be useful if you have to do some kind of dynamic lookup of a UI element, or need to drive pieces of UI that aren’t on the page on initial load.

Here’s an example:

var test_bazThing = function () {
  var nodeId = getSomeDomNodeDynamically();
  windmill.jsTest.actions.click({ id: nodeId });
}

The special “waits.forJS”

This is a wait that depends on the result of arbitrary JavaScript returning `true`. The JavaScript passed can be either a string, or a JavaScript function. Pass the string or function as the `js` property of the params object.

Here’s an example:

test_TempData = { 
  method: "waits.forJS",
  params: { 
    js: function () { 
      return !!windmillMain.tempData;
    }
  }   
};

Grouping tests in arrays

If you have a bunch of actions or functions you want to run in aggregate, you can put them all into an array and the JavaScript test engine will execute them in sequence, but report on their passing/failure as a group. (On failure, execution of the items in the array stops, the failure is reported, and the test engine moves to the next test.)

var test_arrayBarThing = [
  // Click a button
  { method: "click",
    params: {
      id: "fooButton"
    }
  },
  // Wait around a bit for an element on the new page
  { method: "waits.forElement",
    id: {
      'zoomieNode'
    }
  },
  // Verify the Ajaxy update to the page happened
  function () {
    var node = document.getElementById('zoomieNode')
    jum.assertText(node, 'Howdy');
  }
];

JSUnit-Compatible Asserts

Windmill’s JavaScript test framework has the full range of JSUnit-compatible asserts, accessible inside your JavaScript test functions from the “jum” object. Here are the currently available asserts:

assertTrue
assertFalse
assertEquals
assertNotEquals
assertLessThan
assertMoreThan
assertNull
assertNotNull
assertUndefined
assertNotUndefined
assertNaN
assertNotNaN
assertEvaluatesToTrue
assertEvaluatesToFalse
assertContains

The syntax is precisely the same as JSUnit, allowing an optional comment as a first parameter:

jum.assertNotNaN('Totally not a number!', 'Geddy Lee');
=> (Totally not a number!) assertNotNaN -- <Geddy Lee> (String) expected to be a number but was not a number (NaN).
jum.assertNotNaN('Neil Peart');
=> assertNotNaN -- <Neil Peart> (String) expected to be a number but was not a number (NaN).

Namespacing Tests

Windmill JavaScript tests can also be organized into namespaces, just like any other JavaScript code. Namespacing, along with setup/teardown can be used to create sophisticated trees of tests.

Here’s a basic example to give you an idea of what a namespaced test object might look like. The namespace object has a name beginning with “test_”, so the !JavaScript test parser knows it’s a container for tests:

windmill.jsTest.require('shared.js');

test_jsonDom = new function () {
  // Navigate to the main page
  this.setup = windmillMain.shared.test_navMain;

  this.test_getData = function () {
    // Callback function -- sets temp flag to 'returned'
    var success = function (s) {
      var data = eval('('+s+')');
      windmillMain.tempData = data;
    };
    var url = windmillMain.shared.util.getCurrentDir() + 'data.json';
    fleegix.xhr.get(success, url);
  };
  this.test_wait = windmillMain.shared.util.waitForTempData;
  this.test_verifyData = function () {
    var data = windmillMain.tempData;
    var albums = data.albums;
    jum.assertNotNaN(albums.length);
  };
  this.test_writeDom = function () {
    var data = windmillMain.tempData;
    var albums = data.albums;
    var c = $('content');
    for (var i = 0; i < albums.length; i++) {
      var d = $elem('div', {
        id: 'album_' + i,
        innerHTML: albums[i]
      });
      c.appendChild(d);
    }
    jum.assertTrue(c.hasChildNodes());
  };
  this.test_domNodesActionStyle = [
    { method: "asserts.assertNode", params: {id: 'album_9'} },
    { method: "asserts.assertText", params: {id: 'album_9',
      validator: 'Power Windows'} }
  ];
  this.teardown = function () {
    // Clear out the temp data
    windmillMain.tempData = null;
    // Clear the DOM
    $('content').innerHTML = '';
  };
};

The unit tests for the JavaScript test engine provide a lot of examples of namespaced !JavaScript tests.

Running JavaScript Tests

To run your JavaScript tests you can use one of the following methods.

From the Python shell:
run_js_tests('dir/to/js/tests')

From the command line:
windmill firefox http://yourdomain.com jsdir=./js/test/dir

To exit the browser after the tests, include the `exit` option:

windmill firefox http://yourdomain.com jsdir=./js/test/dir exit

Developing Tests — Running Specific Tests or Phases

When you’re working on a test, you don’t want to have to run the setup prerequisites for your new test over and over while you’re trying to debug it. The !JavaScript test engine lets you run only specific tests, or only specific phases (e.g., ‘setup,’ ‘test,’ or ‘teardown’) of a test.

Running only a specific test

To limit to a specific test, in the Windmill Python shell, add the desired test as a second parameter, like so:

run_js_tests("./fleegix_js/tests", 'test_fleegixEvent.test_foo')

Add the `ns:` prefix to limit your run to all the tests in a specific namespace:

run_js_tests("./fleegix_js/tests", 'ns:test_fleegixEvent')

Note: When you’re working with a hierarchy of namespaced tests, setup and teardown phase for a test or test namespace include the setup and teardown for parent namespaces of the test.

Running only a specific phase of your test

To limit your test run to a specific phase, pass the desired phase (or phases in a comma-delimited string), as a third parameter to the `run_js_tests` command. This comes in handy when you’re working on a specific test, and don’t want to go through a full run of setup, test, and teardown every time.

First, get the test ready to run, by running just the setup:
run_js_tests("./fleegix_js/tests", 'ns:test_fleegixXhr', 'setup')

Then, run your the test only, as many times as you need:

run_js_tests("./fleegix_js/tests", 'ns:test_fleegixXhr', 'test')

When you want to run the whole thing, run the teardown once, and re-run the whole shebang:

run_js_tests("./fleegix_js/tests", 'ns:test_fleegixXhr', 'teardown')
run_js_tests("./fleegix_js/tests", 'ns:test_fleegixXhr', 'setup, test, teardown')

Directly from the command line

Limiting to a specific test or phase can also be done when running tests from the command line using the `jsphase` and `jsfilter` options, like so:

windmill firefox http://yourdomain.com jsdir=./js/test/dir jsfilter=ns:test_fleegixXhr jsphase=setup

Testing code which augments native objects

If your code augments native objects (`String`, `Number`, etc.) you will have to change the way your scripts are loaded by Windmill – from using browser’s `eval()` function to script-tag insertion method.

You can do that from command line with `scriptappend` flag:

windmill firefox http://yourdomain.com jsdir=./js/test/dir scriptappend

or in Windmill Python shell by:

windmill.settings['SCRIPT_APPEND_ONLY'] = True

Note that using this method Windmill won’t be able to report syntax errors, but in return you will get a “object test_X cannon’t be found in any file”, thus hinting that you have a syntax error in file which defines that test.

You can aid yourself by using Google’s Closure Compiler which is pretty good at spotting syntax errors.