Parallel tests

[isaacs]

Allow buffered tests to be run concurrently.

Depends on #305.

t.test('first', { parallel: true }, function (t) {
  t.plan(1)
  setTimeout(function () {
    t.pass('this is fine')
  }, 100)
})

t.test('second', { parallel: true }, function (t) {
  t.plan(1)
  setTimeout(function () {
    t.pass('this is fine')
  }, 1)
})

The output, regardless of which test actually runs first, will be:

ok 1 - first {
    1..1
    ok 1 - this is fine
}

ok 2 - second {
    1..1
    ok 1 - this is fine
}

Do this by making parallel tests be buffered (using whatever mechanism is built for #305), and just kick them off right away, regardless of whether or not there's a this._currentChild. When encountered in the queue, if they're finished, then their buffered output is handled. If they aren't, then the process waits for them to finish before proceeding.

What this will mean is that a parallel: true subtest won't wait for non-parallel tests, but non-parallel tests will wait for a parallel test that might precede it.

This will be particularly useful for running a bunch of tests that all have to hit a server, and then following it up with a non-parallel test to close down the server once they're all done.

t.test('setup server', function (t) {
  server.listen(3000, function () {
    t.end()
  })
})

t.test('check all the routes', function (t) {
  t.test('login flow', { parallel: true }, function (t) {
    // bunch of child tests here that are _not_ parallel, running in series
    // but this series runs alongside the other series.
    t.test('create account', function (t) { ... t.end() })
    t.test('login as new account', function (t) { .. t.end() .. })
    t.test('delete account', function (t) { ... t.end() ... })
    t.end()
  })

  t.test('documentation pages', { parallel: true }, function (t) {
    // check documentation exists.
    t.end()
  })

  // maybe other files with a bunch more tests
  t.spawn(process.execPath, [__dirname + '/test/other-routes.js'], {}, 'other routes', { parallel: true })
  t.spawn(process.execPath, [__dirname + '/test/the-css.js'], {}, 'css tests', { parallel: true })

  // .. other things

  // now the NOT-parallel thing
  t.test('close server', function (t) {
    // if there are hanging connections, this will time out
    server.on('close', t.end)
    server.close()
  })

  t.end()
})

[isaacs]

Work happening on v10 branch.

Todo List:

• return Promises from test/stdin/spawn
• handle case where test() ends before promise resolution
• time outs! (Including handling SIGTERM from a parent proc.)
• root test runner exported as the package main! (kinda important)
• autoend (mostly for the root test runner)
• emit the version
• ignore EPIPE errors
• How to do process.on('uncaughtException') now that we can't easily dive into the child test levels? (Especially since multiple children may be running at once!) Maybe threw() should just check for a non-buffered test in process, and if not there, then error at the top and let it be bad tap? Seems klunky, especially given the existence of assertion libs that do this.
• Don't bother running enqueued child tests if there's a bailout! Probably at that point, it's worth dropping the whole queue?
• Closeout and fail if there are unfinished tests in the queue at exit
• treat a slow synchronous end() as a timeout (ie, if end comes after the timer would have expired)
• some way to turn the timer off in the runner bin --no-timeout or some such
• beforeEach, afterEach, teardown
• mocha dsl
• how to handle t.current()? Deprecate? (Yes, deprecate 73190d6)
• report test timing
• docs

[isaacs]

Only one checkbox left, but it's a pretty big one.

[ORESoftware]

@isaacs I like how "add domains" was carefully omitted from the list :) or maybe you haven't added them yet? It's a fairly big change :) I am a fan of domains and would like for more people to admit they are useful.

nope, definitely see some domain use going on in the codebase :) welcome to the club. A much better pattern than the one that Mocha uses.

[pke]

Any progress on that?

[isaacs]

@pke http://www.node-tap.org/parallel/