Controlling Parallelism

[ccorcos]

I looked up "nested tests" and found this issue but that discussion was going a different direction from what I was thinking.

I'm working on a CLI tool project and I want to convert my test from that shells script to Node.js, preferably with Ava. However, some things need to happen in a certain order.

Here's how far I've gotten:

import test from 'ava'
import shell from 'shelljs'
import exists from 'doug/utils/exists'

shell.config.fatal = true

test.before('doug-app init', (t) => {
  // initialize doug-app
  shell.cd('~')
  shell.exec('doug-app init doug-app-test')
  t.truthy(exists('doug-app-test'))

  // link doug-app
  shell.cd('cd ~/doug-app-test')
  shell.exec('npm link doug-app')

  // setup local git origin
  shell.mkdir('doug-app-origin')
  shell.cd('doug-app-origin')
  shell.exec('git init --bare')

  // push initial commit
  shell.cd('~/doug-app-test')
  shell.exec([
    'git init',
    'git add .',
    'git config --global user.email "test@test.com"',
    'git config --global user.name "Doug Test"',
    'git commit -m "doug-app-test"',
    'git remote add origin ~/doug-app-origin',
    'git push origin master',
  ].join('; '))
})

test.test('doug-app test', (t) => {
  shell.cd('cd ~/doug-app-test')
  shell.exec('doug-app test')
})

test.test('doug-app build', (t) => {
  shell.cd('cd ~/doug-app-test')
  shell.exec('doug-app build')
  t.truthy(exists('dist'))
})

test.test('doug-app deploy', (t) => {
  shell.cd('cd ~/doug-app-test')
  shell.exec('doug-app deploy')
  shell.cd('~/doug-app-origin')
  t.truthy(shell.exec('git branch | grep gh_pages').stdout.trim())
})

The thing is, deploy need to happen after build. But it is a different test, and can in parallel with other commands such as test...

[novemberborn]

Ideally tests are independent from each other. If that's not feasible you could use test.serial(). That ensures that tests, within a test file, execute in the order in which they were declared.

[ccorcos]

Only certain parts need to be run serially, but a lot of things can run in parallel. We get some control over that with before and after. It just seems like with the recursive nature of the way Ava works, this would be a cheap win to get a very flexible way of controlling the parallelism of your tests.

Consider something like this where we have fine-grained control over what's in parallel and whats serial.

test.test('doug-app init', (t) => {
  // initialize doug-app
  shell.cd('~')
  shell.exec('doug-app init doug-app-test')
  t.truthy(exists('doug-app-test'))

  t.before('link doug-app', (t) => {
    // link doug-app
    shell.cd('cd ~/doug-app-test')
    shell.exec('npm link doug-app')
  })

  t.test('doug-app test', (t) => {
    shell.cd('cd ~/doug-app-test')
    shell.exec('doug-app test')
  })

  t.test('doug-app build', (t) => {

    shell.cd('cd ~/doug-app-test')
    shell.exec('doug-app build')
    t.truthy(exists('dist'))

    t.before('setup git origin' (t) => {
      // setup local git origin
      shell.mkdir('doug-app-origin')
      shell.cd('doug-app-origin')
      shell.exec('git init --bare')

      // push initial commit
      shell.cd('~/doug-app-test')
      shell.exec([
        'git init',
        'git add .',
        'git config --global user.email "test@test.com"',
        'git config --global user.name "Doug Test"',
        'git commit -m "doug-app-test"',
        'git remote add origin ~/doug-app-origin',
        'git push origin master',
      ].join('; '))
    })

    t.test('doug-app deploy', (t) => {
      shell.cd('cd ~/doug-app-test')
      shell.exec('doug-app deploy')
      shell.cd('~/doug-app-origin')
      t.truthy(shell.exec('git branch | grep gh_pages').stdout.trim())
    })    
  })
})

[novemberborn]

AVA's philosophy is that tests should be independent of each other, and indeed should be able to run in any order. Sometimes tests require exclusive access to a particular resource, in which case you should use t.serial. Sometimes that resource is so exclusive you cannot even run test files in parallel, for which we have a global serial option.

Regardless of where we end up with grouping tests the above will hold. The kind of test orchestration you're looking for is beyond the scope of AVA.

That said, AVA is pretty good with promises, and since by default all tests are started at the same time you could build a library on top of AVA which achieves the sequencing you're looking for.

[ccorcos]

That said, AVA is pretty good with promises, and since by default all tests are started at the same time you could build a library on top of AVA which achieves the sequencing you're looking for.

I can, and I've made it work by basically just having one giant test. But ideally, I could specify checkpoints in there so AVA can report 5 tests pass rather than just 1 and if there's a test failure, report exactly which test failed...

[novemberborn]

You could have promises for the checkpoints, and then place relevant assertions in separate tests.

[ccorcos]

I see. So something like this?

const checkpoint = () => {
  let resolve = undefined
  const promise = new Promise((r) => resolve = r)
  promise.resolve = resolve
  return promise
}

const build = checkpoint()

test('build the project', () => {
  return buildProject().then(build.resolve)
})

test('deploy the project', () => {
  return build.then(deployProject)
})

[novemberborn]

Yup. Or perhaps do all that setup in a test.before() and assign a promise for each stage to a variable. Then in your tests you could write const result = await buildStage and then do the assertions.

[ccorcos]

I like that a lot actually. Thanks @novemberborn!