setup() methods are called prematurely when tests are loaded asynchronously

[changemewtf]

I believe this might be an example of this problem in the wild.

If tests are loaded asynchronously, the document is loaded before they are read, which changes the values of config.autorun and config.blocking and eventually causes the callbacks queued in Test.queue() to be run out-of-sequence.

This pull request contains a separate test HTML file with accompanying JS tests that exemplify the problem.

The exact sequence of problematic events goes like this:

1. DOM loads
2. QUnit.load() completes, and calls QUnit.start()
3. QUnit.start() sets config.blocking to false, then calls process()
4. There's nothing in the queue, so process() calls done()
5. done() sets config.autorun to true
6. Test A queues init(), which is autorun by synchronize()
7. Test A queues the inner run(), which is autorun
8. Test A queues setup(), which is autorun
9. Test A queues run(), which is autorun; now we're blocking.
10. Test A queues teardown() and finish(). queue is now [A.teardown, A.finish]
11. Test B queues init() and inner run(). queue is now [A.teardown, A.finish, B.init, B.run]
12. Test C queues init() and inner run(). queue is now [A.teardown, A.finish, B.init, B.run, C.init, C.run]
13. Test A's teardown() and finish() are run; no more blocking. queue is now [B.init, B.run, C.init, C.run]
14. Test B's init() and inner run() are run. queue is now [C.init, C.run]
15. Test B queues setup(). queue is now [C.init, C.run, B.setup]
16. synchronize() queues Test B's setup() and kicks off process()
17. process() runs the next functions in the queue, which are Test C's init() and run().
18. Test C queues setup(). queue is now [B.setup, C.setup]

And now you're thinking with portals.

[pscheit]

nice job, good test though, i had the same idea with the array for the stack exchange post but yours is more elegant.

[jzaefferer]

Usually what you need to do here is to set QUnit.config.autorun = false, then call QUnit.start() once you're done loading tests. Certainly needs (better) documentation, but I don't think there's much the framework could do.

[jzaefferer]

There's an example for using QUnit.config.autostart in the API documentation now: http://api.qunitjs.com/QUnit.config/

Beyond that I would need more input to address this, until then I've got to assume the above is a valid solution.

[pscheit]

thanks, that solved it. I'm glad that you have now a real api documentation.

[changemewtf]

@jzaefferer - How about adding QUnit.config.autostart to the file in my test? I don't think there's already a test for this functionality (is there?)

[jzaefferer]

@mcantor good suggestion, thanks. Landed in 4e03a4b