test_runner: support test plans

[marco-ippolito]

I picked up the work from this comment #44125 (comment) by @cjihrig to add support for test planning

[nodejs-github-bot]

Review requested:

• @nodejs/test_runner

[RedYetiDev]

IMO the t.plan() should also be settable via the test options

[MoLow]

Can you please elaborate on the use cases of this feature?
We already error if a test never ends and the event loop is drained - in what scenario should I worry not all assertions will be checked?
and what makes assertions different than any other piece of code inside the test?

[marco-ippolito]

Can you please elaborate on the use cases of this feature? We already error if a test never ends and the event loop is drained - in what scenario should I worry not all assertions will be checked? and what makes assertions different than any other piece of code inside the test?

It makes sure that an assertion is called inside a callback, when the callback might not be invoked.
This feature is actually common in a lot of testing libraries such as

• tap https://node-tap.org/basics/#planned-assertion-counts
• jest https://jestjs.io/docs/expect#expectassertionsnumber
• vitest https://vitest.dev/api/expect.html#expect-assertions

test('make sure callback is invoked', (t) => {
 t.plan(1)
  server.listen(0, (r) => {
   // assert something in here
   t.ok(true)
  });
});

With plan you are sure than the callback was invoked and the assertion was performed, otherwise the test would pass anyways.

[MoLow]

I am -0.5 on adding this.
if one wants to assert his code run they can use mock.fn/mock.method and assert it was actually called. I don't think assertions are very different in that aspect

[simoneb]

Hey folks, Marco is working with me and I suggested that he works on this feature that is missing from the Node.js test runner and it's necessary in certain specific cases.

The use case is fairly specific and it pertains callback-based APIs. When the callback is called and you need to make assertions there, there is no easy way to do that except telling the test that it should not end before X number of assertions have been made (bar a test timeout).

Let's take this example from the Fastify docs:

const tap = require('tap')
const buildFastify = require('./app')

tap.test('GET `/` route', t => {
  t.plan(4)

  const fastify = buildFastify()

  // At the end of your tests it is highly recommended to call `.close()`
  // to ensure that all connections to external services get closed.
  t.teardown(() => fastify.close())

  fastify.inject({
    method: 'GET',
    url: '/'
  }, (err, response) => {
    t.error(err)
    t.equal(response.statusCode, 200)
    t.equal(response.headers['content-type'], 'application/json; charset=utf-8')
    t.same(response.json(), { hello: 'world' })
  })
})

Let's forget for a second that fastify.inject also has a promise API. Without t.plan(4) you have no easy way to make sure that the assertions in the callback have been invoked. E.g. if inject due to a bug didn't invoke the callback, this test would pass just fine if you didn't have a way to tell the test that you're expecting 4 assertions.

I hope this clarifies why this feature is necessary.

[mcollina]

I think this feature is needed.

@marco-ippolito can you add a test that verifies what happens if there were more assertion than the plan?

[mcollina]

This is missing a way for other assertion frameworks to tap into the test planning. If I recall correctly, the need for this was what stopped @cjihrig working on this feature.

[marco-ippolito]

@cjihrig for some reason it squashed the commit and set myself as author, I wanted to give you full credit to you on this
Amended in 0c83f80

[targos]

This PR adds a feature so it should be labeled semver-minor PRs that contain new features and should be released in the next minor version. (this is not supposed to be done by releasers).