testharness.js provides a framework for writing testcases. It is intended to provide a convenient API for making common assertions, and to work both for testing synchronous and asynchronous DOM features in a way that promotes clear, robust, tests.
The test harness script can be used from HTML or SVG documents and web worker scripts.
From an HTML or SVG document, start by importing both testharness.js
and
testharnessreport.js
scripts into the document:
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
Refer to the Web Workers section for details and an example on testing within a web worker.
Within each file one may define one or more tests. Each test is atomic in the
sense that a single test has a single result (PASS
/FAIL
/TIMEOUT
/NOTRUN
).
Within each test one may have a number of asserts. The test fails at the first
failing assert, and the remainder of the test is (typically) not run.
If the file containing the tests is a HTML file, a table containing the test
results will be added to the document after all tests have run. By default this
will be added to a div
element with id=log
if it exists, or a new div
element appended to document.body
if it does not.
NOTE: By default tests must be created before the load event fires. For ways to create tests after the load event, see "Determining when all tests are complete", below.
To create a synchronous test use the test() function:
test(test_function, name, properties)
test_function
is a function that contains the code to test. For example a
trivial passing test would be:
test(function() {assert_true(true)}, "assert_true with true")
The function passed in is run in the test()
call.
properties
is a javascript object for passing extra options to the
test. Currently it is only used to provide test-specific
metadata, as described in the metadata section below.
Testing asynchronous features is somewhat more complex since the result of a test may depend on one or more events or other callbacks. The API provided for testing these features is indended to be rather low-level but hopefully applicable to many situations.
To create a test, one starts by getting a Test object using async_test:
async_test(name, properties)
e.g. var t = async_test("Simple async test")
Assertions can be added to the test by calling the step method of the test object with a function containing the test assertions:
t.step(function() {assert_true(true)});
When all the steps are complete, the done() method must be called:
t.done();
As a convenience, async_test can also takes a function as first argument.
This function is called with the test object as both its this
object and
first argument. The above example can be rewritten as:
async_test(function(t) {
object.some_event = function() {
t.step(function (){assert_true(true); t.done();});
};
}, "Simple async test");
which avoids cluttering the global scope with references to async tests instances.
The properties argument is identical to that for test()
.
In many cases it is convenient to run a step in response to an event or a callback. A convenient method of doing this is through the step_func method which returns a function that, when called runs a test step. For example
object.some_event = t.step_func(function(e) {assert_true(e.a)});
For asynchronous callbacks that should never execute, unreached_func
can
be used. For example:
object.some_event = t.unreached_func("some_event should not fire");
Keep in mind that other tests could start executing before an Asynchronous Test is finished.
promise_test
can be used to test APIs that are based on Promises:
promise_test(test_function, name, properties)
test_function
is a function that receives a test as an argument and returns a
promise. The test completes when the returned promise resolves. The test fails
if the returned promise rejects.
E.g.:
function foo() {
return Promise.resolve("foo");
}
promise_test(function() {
return foo()
.then(function(result) {
assert_equals(result, "foo", "foo should return 'foo'");
});
}, "Simple example");
In the example above, foo()
returns a Promise that resolves with the string
"foo". The test_function
passed into promise_test
invokes foo
and attaches
a resolve reaction that verifies the returned value.
Note that in the promise chain constructed in test_function
assertions don't
need to wrapped in step
or step_func
calls.
Unlike Asynchronous Tests, Promise Tests don't start running until after the previous Promise Test finishes.
promise_rejects
can be used to test Promises that need to reject:
promise_rejects(test_object, code, promise)
The code
argument is equivalent to the same argument to the assert_throws
function.
Here's an example where the bar()
function returns a Promise that rejects
with a TypeError:
function bar() {
return Promise.reject(new TypeError());
}
promise_test(function(t) {
return promise_rejects(t, new TypeError(), bar);
}, "Another example");
EventWatcher
is a constructor function that allows DOM events to be handled
using Promises, which can make it a lot easier to test a very specific series
of events, including ensuring that unexpected events are not fired at any point.
Here's an example of how to use EventWatcher
:
var t = async_test("Event order on animation start");
var animation = watchedNode.getAnimations()[0];
var eventWatcher = new EventWatcher(watchedNode, ['animationstart',
'animationiteration',
'animationend']);
eventWatcher.wait_for(t, 'animationstart').then(t.step_func(function() {
assertExpectedStateAtStartOfAnimation();
animation.currentTime = END_TIME; // skip to end
// We expect two animationiteration events then an animationend event on
// skipping to the end of the animation.
return eventWatcher.wait_for(['animationiteration',
'animationiteration',
'animationend']);
})).then(t.step_func(function() {
assertExpectedStateAtEndOfAnimation();
test.done();
}));
wait_for
either takes the name of a single event and returns a Promise that
will resolve after that event is fired at the watched node, or else it takes an
array of the names of a series of events and returns a Promise that will
resolve after that specific series of events has been fired at the watched node.
EventWatcher
will assert if an event occurs while there is no wait_for
()
created Promise waiting to be fulfilled, or if the event is of a different type
to the type currently expected. This ensures that only the events that are
expected occur, in the correct order, and with the correct timing.
Sometimes, particularly when dealing with asynchronous behaviour,
having exactly one test per page is desirable, and the overhead of
wrapping everything in functions for isolation becomes
burdensome. For these cases testharness.js
support "single page
tests".
In order for a test to be interpreted as a single page test, the
it must simply not call test()
or async_test()
anywhere on the page, and
must call the done()
function to indicate that the test is complete. All
the assert_*
functions are avaliable as normal, but are called without
the normal step function wrapper. For example:
<!doctype html>
<title>Example single-page test</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<body>
<script>
assert_equals(document.body, document.getElementsByTagName("body")[0])
done()
</script>
The test title for sinple page tests is always taken from document.title
.
Functions for making assertions start assert_
. The full list of
asserts avaliable is documented in the asserts section
below. The general signature is
assert_something(actual, expected, description)
although not all assertions precisely match this pattern e.g. assert_true
only takes actual
and description
as arguments.
The description parameter is used to present more useful error messages when a test fails
NOTE: All asserts must be located in a test()
or a step of an
async_test()
, unless the test is a single page test. Asserts outside
these places won't be detected correctly by the harness and may cause
unexpected exceptions that will lead to an error in the harness.
Occasionally tests may create state that will persist beyond the test itself.
In order to ensure that tests are independent, such state should be cleaned
up once the test has a result. This can be achieved by adding cleanup
callbacks to the test. Such callbacks are registered using the add_cleanup
function on the test object. All registered callbacks will be run as soon as
the test result is known. For example
test(function() {
window.some_global = "example";
this.add_cleanup(function() {delete window.some_global});
assert_true(false);
});
In general the use of timeouts in tests is discouraged because this is an observed source of instability in real tests when run on CI infrastructure. In particular if a test should fail when something doesn't happen, it is good practice to simply let the test run to the full timeout rather than trying to guess an appropriate shorter timeout to use.
In other cases it may be necessary to use a timeout (e.g., for a test
that only passes if some event is not fired). In this case it is
not permitted to use the standard setTimeout
function. Instead one
must use the step_timeout
function:
var t = async_test("Some test that does something after a timeout");
t.step_timeout(function() {assert_true(true); this.done()}, 2000);
The difference between setTimeout
and step_timeout
is that the
latter takes account of the timeout multiplier when computing the
delay; e.g., in the above case a timeout multiplier of 2 would cause a
pause of 4000ms before calling the callback. This makes it less likely
to produce unstable results in slow configurations.
For single-page tests, step_timeout
is also available as a global
function.
The overall harness admits two timeout values "normal"
(the
default) and "long"
, used for tests which have an unusually long
runtime. After the timeout is reached, the harness will stop
waiting for further async tests to complete. By default the
timeouts are set to 10s and 60s, respectively, but may be changed
when the test is run on hardware with different performance
characteristics to a common desktop computer. In order to opt-in
to the longer test timeout, the test must specify a meta element:
<meta name="timeout" content="long">
Occasionally tests may have a race between the harness timing out and
a particular test failing; typically when the test waits for some event
that never occurs. In this case it is possible to use test.force_timeout()
in place of assert_unreached()
, to immediately fail the test but with a
status of TIMEOUT
. This should only be used as a last resort when it is
not possible to make the test reliable in some other way.
Sometimes tests require non-trivial setup that may fail. For this purpose
there is a setup()
function, that may be called with one or two arguments.
The two argument version is:
setup(func, properties)
The one argument versions may omit either argument.
func is a function to be run synchronously. setup()
becomes a no-op once
any tests have returned results. Properties are global properties of the test
harness. Currently recognised properties are:
explicit_done
- Wait for an explicit call to done() before declaring all
tests complete (see below; implicitly true for single page tests)
output_document
- The document to which results should be logged. By default
this is the current document but could be an ancestor document in some cases
e.g. a SVG test loaded in an HTML wrapper
explicit_timeout
- disable file timeout; only stop waiting for results
when the timeout()
function is called (typically for use when integrating
with some existing test framework that has its own timeout mechanism).
allow_uncaught_exception
- don't treat an uncaught exception as an error;
needed when e.g. testing the window.onerror
handler.
timeout_multiplier
- Multiplier to apply to per-test timeouts.
By default the test harness will assume there are no more results to come when:
- There are no
Test
objects that have been created but not completed - The load event on the document has fired
This behaviour can be overridden by setting the explicit_done
property to
true in a call to setup()
. If explicit_done
is true, the test harness will
not assume it is done until the global done()
function is called. Once done()
is called, the two conditions above apply like normal.
Dedicated and shared workers don't have an event that corresponds to the load
event in a document. Therefore these worker tests always behave as if the
explicit_done
property is set to true. Service workers depend on the
install
event which is fired following the completion of running the
worker.
There are scenarios in which is is desirable to create a large number of
(synchronous) tests that are internally similar but vary in the parameters
used. To make this easier, the generate_tests
function allows a single
function to be called with each set of parameters in a list:
generate_tests(test_function, parameter_lists, properties)
For example:
generate_tests(assert_equals, [
["Sum one and one", 1+1, 2],
["Sum one and zero", 1+0, 1]
])
Is equivalent to:
test(function() {assert_equals(1+1, 2)}, "Sum one and one")
test(function() {assert_equals(1+0, 1)}, "Sum one and zero")
Note that the first item in each parameter list corresponds to the name of the test.
The properties argument is identical to that for test()
. This may be a
single object (used for all generated tests) or an array.
The framework provides callbacks corresponding to 4 events:
start
- triggered when the first Test is createdtest_state
- triggered when a test state changesresult
- triggered when a test result is recievedcomplete
- triggered when all results are recieved
The page defining the tests may add callbacks for these events by calling the following methods:
add_start_callback(callback)
- callback called with no arguments
add_test_state_callback(callback)
- callback called with a test argument
add_result_callback(callback)
- callback called with a test argument
add_completion_callback(callback)
- callback called with an array of tests
and an status object
tests have the following properties:
-
status
- A status code. This can be compared to thePASS
,FAIL
,TIMEOUT
andNOTRUN
properties on the test object -
message
- A message indicating the reason for failure. In the future this will always be a string
The status object gives the overall status of the harness. It has the following properties:
-
status
- Can be compared to theOK
,ERROR
andTIMEOUT
properties -
message
- An error message set when the status isERROR
In order to collect the results of multiple pages containing tests, the test harness will, when loaded in a nested browsing context, attempt to call certain functions in each ancestor and opener browsing context:
- start -
start_callback
- test_state -
test_state_callback
- result -
result_callback
- complete -
completion_callback
These are given the same arguments as the corresponding internal callbacks described above.
Where supported, the test harness will also send messages using cross-document messaging to each ancestor and opener browsing context. Since it uses the wildcard keyword (*), cross-origin communication is enabled and script on different origins can collect the results.
This API follows similar conventions as those described above only slightly
modified to accommodate message event API. Each message is sent by the harness
is passed a single vanilla object, available as the data
property of the event
object. These objects are structures as follows:
- start -
{ type: "start" }
- test_state -
{ type: "test_state", test: Test }
- result -
{ type: "result", test: Test }
- complete -
{ type: "complete", tests: [Test, ...], status: TestsStatus }
The testharness.js
script can be used from within dedicated workers, shared
workers and service
workers.
Testing from a worker script is different from testing from an HTML document in several ways:
-
Workers have no reporting capability since they are runing in the background. Hence they rely on
testharness.js
running in a companion client HTML document for reporting. -
Shared and service workers do not have a unique client document since there could be more than one document that communicates with these workers. So a client document needs to explicitly connect to a worker and fetch test results from it using
fetch_tests_from_worker
. This is true even for a dedicated worker. Once connected, the individual tests running in the worker (or those that have already run to completion) will be automatically reflected in the client document. -
The client document controls the timeout of the tests. All worker scripts act as if they were started with the
explicit_timeout
option (see the Harness timeout section). -
Dedicated and shared workers don't have an equivalent of an
onload
event. Thus the test harness has no way to know when all tests have completed (see Determining when all tests are complete). So these worker tests behave as if they were started with theexplicit_done
option. Service workers depend on the oninstall event and don't require an explicitdone
call.
Here's an example that uses a dedicated worker.
worker.js
:
importScripts("/resources/testharness.js");
test(function(t) {
assert_true(true, "true is true");
}, "Simple test");
// done() is needed because the testharness is running as if explicit_done
// was specified.
done();
test.html
:
<!DOCTYPE html>
<title>Simple test</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<div id="log"></div>
<script>
fetch_tests_from_worker(new Worker("worker.js"));
</script>
The argument to the fetch_tests_from_worker
function can be a
Worker
,
a SharedWorker
or a ServiceWorker
.
Once called, the containing document fetches all the tests from the worker and
behaves as if those tests were running in the containing document itself.
asserts that actual
is strictly true
asserts that actual
is strictly false
asserts that actual
is the same value as expected
asserts that actual
is a different value to expected
.
This means that expected
is a misnomer.
asserts that expected
is an Array, and actual
is equal to one of the
members i.e. expected.indexOf(actual) != -1
asserts that actual
and expected
have the same
length and the value of each indexed property in actual
is the strictly equal
to the corresponding property value in expected
asserts that actual
is a number within +-
epsilonof
expected`
asserts that actual
is a number less than expected
asserts that actual
is a number greater than expected
asserts that actual
is a number between lower
and upper
but not
equal to either of them
asserts that actual
is a number less than or equal to expected
asserts that actual
is a number greater than or equal to expected
asserts that actual
is a number between lower
and upper
or
equal to either of them
asserts that actual
matches the regexp expected
asserts that the class string of object
as returned in
Object.prototype.toString
is equal to class_name
.
assert that object has own property property_name
assert that object does not have an own property named
property_name
but that property_name
is present in the prototype
chain for object
assert that an object that is an instance of some interface has the attribute attribute_name following the conditions specified by WebIDL
assert that property property_name
on object is readonly
code
- the expected exception. This can take several forms:
- string - the thrown exception must be a DOMException with the given name, e.g., "TimeoutError" (for compatibility with existing tests, a constant is also supported, e.g., "TIMEOUT_ERR")
- object - the thrown exception must have a property called "name" that matches code.name
- null - allow any exception (in general, one of the options above should be used)
func
- a function that should throw
asserts if called. Used to ensure that some codepath is not taken e.g. an event does not fire.
asserts that one assert_func(actual, expected_array_N, extra_arg1, ..., extra_arg_N)
is true for some expected_array_N
in expected_array
. This only works for assert_func
with signature assert_func(actual, expected, args_1, ..., args_N)
. Note that tests
with multiple allowed pass conditions are bad practice unless the spec specifically
allows multiple behaviours. Test authors should not use this method simply to hide
UA bugs.
deprecated
asserts that object has an own property property_name
deprecated
assert that object does not have own property property_name
It is possible to add optional metadata to tests; this can be done in
one of two ways; either by adding <meta>
elements to the head of the
document containing the tests, or by adding the metadata to individual
[async_]test
calls, as properties.