Async tests can abruptly stop in standalone Node.js mode

[getify]

I'm not sure if this behavior is by-design, or if there's just no way for QUnit to fix it, but this situation just bit me and it was very confusing.

Here's my minimal test case:

"use strict";

var QUnit = require("qunit");

QUnit.done(function done(results) {
	console.log("done:",results);
});

QUnit.test("temp",function(assert){
	var done = assert.async();
	assert.expect(1);

	setTimeout(function(){
		console.log("timer!");
	},1000);
});

QUnit.test( "temp2", function(assert){
	assert.expect(1);
	assert.ok(true,"temp2");
});

QUnit.start();

And when I run this:

$] node tmp.js
timer!
$]

In other words, the first test case fires the timer, but since it never runs the done(), it just silently dies. Since node doesn't have any other waiting event handlers, node just exits. The test suite doesn't fire my completion handler to notify me that a test failed to meet its assertion, and it doesn't try to run the second test at all. It just stops.

And even worse, the test suite exits with a zero exit code, meaning my other CLI tools consider this a passing test suite run. :(

This seems like broken behavior to me. Shouldn't there be some sort of timeout in the background for async tests, or something like that?

Or maybe qunit could register a process.on("exit", ..) handler, that if the test suite hasn't completed by the time the process tries to finish, the handler forces a non-zero exit code to signal failure to the CLI, and emits some error about an abnormally terminated test suite?

[getify]

Should mention: I'm using v2.3.0 and node 7.7.4.

[platinumazure]

It definitely feels like a bug to me. On the browser, the tests would just hang indefinitely and the run would not be considered a passing run.

I like the idea of using process.on("exit") to detect abnormal termination, since we should be able to track if all registered tests have run or not.

Regarding timeouts, I wouldn't want to implement one by default but I would be open to an option for the same. I believe we have per-test timeouts supported in the browser, but I don't know if the CLI supports it yet.

@trentmwillis Any thoughts on this?

[getify]

Regarding timeouts, I wouldn't want to implement one by default but I would be open to an option for the same.

Similar to assert.expect(..), it would be useful if assert.timeout(..) could be called to provide a timeout delay, after which the test would be considered a failure.

On a related topic, assert.failure() would be useful to proactively mark the test as failed, so that I could do my own timeouts... like setTimeout( assert.failure, 1000 ) for example.

[leobalter]

that's right. QUnit is not holding node until the tests reach completion.

[platinumazure]

@leobalter Looks like the CLI should be handling this via these lines in bin/run.js. But that code is not invoked when the Node API is used via require("qunitjs").

I'm not really sure it should be up to QUnit core to handle this (it should be runtime-agnostic). I think if someone is using the Node API, maybe it should be up to them to listen for the runEnd event themselves... What do you think? (Another option would be to just register a noop function in a Node-specific wrapper around core, but...)

EDIT: On second thought, I think the correct solution is to create a Node wrapper which basically inherits from the core QUnit object. And it should use something like QUnit.on( "runEnd", function() { process.nextTick( process.exit ); }); to hold the event loop until the test is done. We can augment with timeout events later to allow some conditional forced exiting.

[leobalter]

I think if someone is using the Node API, maybe it should be up to them to listen for the runEnd event themselves

That's how we handled it in our grunt task.

What do you think? Another option would be to just register a noop function in a Node-specific wrapper around core, but...

Now I wonder if we would break previous implementations by "fixing" this.

[trentmwillis]

I'll be looking into this, planning on doing several fixes/improvements related to assert.async.

[trentmwillis]

So there are a couple parts to this issue:

1. Currently, we have QUnit.config.testTimeout which should rectify the issue here, however it is undefined by default. I think this should be set to some reasonable default (60 seconds maybe?), though this might be considered a breaking change.
2. In the case where QUnit.config.testTimeout is set to undefined (or any falsy value), we should check for early exits of the process. We can do this in our CLI implementation, relatively easily I suspect.
3. Related to (2), I think the CLI should likely default to exiting with a non-zero status code, unless the test suite has finished successfully, this will help us avoid false-positives.
4. There is no easy way to set a per-test timeout. I think we should implement a assert.timeout() method.

[puppetmaster3]

+1. I need a good async/promise testing for fetch()

[getify]

Any update on this?