Supporting asynchronous tests

[juandopazo]

In order to test module loaders and promises, test262 needs to support asynchronous tests.

The most straightforward way would be to extend the signature of runTestCase(fn) The test case function could get a done function as a parameter, similar to what Mocha does.

I think this would introduce the least increase to the API surface.

An open issue is what to do with tests that time out, because test262 should not know about setTimeout.

@domenic @kriskowal would you mind commenting on this?

[kriskowal]

It is certainly the norm for test harnesses to pass a done(?error) method and require it to be called by the test only if testFn.length >= 1. The pattern composes well and addresses the needs of testing promises themselves. For higher layers, it is useful if the scaffold will check whether testFn returns an object, and if so, treat it as a promise for the completion of the test.

[bterlson]

ping @anba, who may have an opinion on augmenting runTestCase further...

[anba]

Mocha uses a setTimeout based solution for its asynchronous tests, so, as already mentioned in #comment 0, this approach will not work for test262. It's kind of difficult to properly test Promise support in ECMAScript implementations, because task queues are a completely new concept in ECMAScript and there are no existing tools to track eventual computations. For example the JavaScriptCore Promise tests rely on comparing console output. And the V8 tests use Object.observe() and some internal functions.

My first idea was to use the @negative attribute and hope implementations use the same uncaught exception handling in NextTask as for other uncaught exceptions, that means to stop script execution. Unfortunately the V8 implementation silently ignores exceptions in NextTask 😞.

/**
 * @negative TaskExecuted
 */

var TaskExecuted = new Error("TaskExecuted");

// Function to throw an exception from within "8.4.2 NextTask"
function stopExecution() {
  let p = Promise.reject();
  p.constructor = function(r) {
    r(() => {}, () => { throw TaskExecuted });
  };
  p.then();
}

try {
  Promise.resolve().then(stopExecution);
} catch (e) {
  // catch any error here to ensure `throw` from stopExecution()
  // is executed after current turn
}

Related:

• @negative attribute: https://mail.mozilla.org/pipermail/test262-discuss/2013-September/000242.html
• http://wiki.ecmascript.org/doku.php?id=test262:test_case_format#informal_test262_test_case_semantics
• https://bugs.ecmascript.org/show_bug.cgi?id=1716

[juandopazo]

I'l start sketching the promise tests as if runTestCase took a function argument with a done optional parameter. I'll probably have them done quite a bit after we figure out how to deal with tasks.

Clarifying:

runTestCase(function () {
 // looks at the return value
});

runTestCase(function (done) {
  // waits for `done` to be called
  // If `done` is called with a non-undefined first parameter
  // it is considered the same as calling $ERROR
});

[bterlson]

I'm not sure I would prefer runTestCase over something like

/**
 * @async
 */

doSomethingAsync().then(function() {
    $DONE();
});

Where the harness will wait for $DONE to be called when the @async attribute is present, with some timeout value (implementing timeout without setTimeout is likely difficult but possible I believe).

I say this because I think the current guidance is to effectively deprecate runTestCase, so it would be a shame to resurrect it for async if this is true.

[juandopazo]

I'd be ok with that too, though I find it a bit weird to be using promises to test promises. It seems like simplicity should win here. Also promises may catch unexpected errors.

[bterlson]

The only harness requirement is $DONE function and support for @async attribute. In the example above, consider it a test for "promise invokes .then handler" or something... IOW I think all the promise machinery is under test. I actually think the two proposed implementations are mostly identical, the only real difference being whether $DONE/done are globally exposed or passed as a param to the runTestCase callback. @juandopazo, is that your understanding as well?

[juandopazo]

Oh I got it. My thought was that having done as a parameter would communicate by itself that the test is async without needing the async attribute. But I'm ok either way.

[bterlson]

I suppose the harness could search for $DONE in the test case somewhere to know that it's async... probably a decent idea, but I haven't thought too hard about it :)

[domenic]

This was fixed by #34

[smikes]

I believe this is fixed now, and there is now async test support in:

• python console runner
• node test262-harness console runner
• browser runner

Comments, @bterlson @juandopazo

[bterlson]

Agreed, closing this.