test runner tap output incorrectly reports subtests

[isaacs]

Version

20.6.1

Platform

Darwin moxy.lan 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

Subsystem

test

What steps will reproduce the bug?

// test.mjs
import { test } from 'node:test'
import assert from 'node:assert'

test('add tests', async t => {
  assert.equal(1, 1, 'this is equal')
  await t.test('adding strings', () => {
    assert.equal('a' + 'b', 'ab', 'concatenation')
  })
  await t.test('adding numbers', () => {
    assert.equal(1 + 2, 3, 'addition')
  })
})

Output:

TAP version 13
# Subtest: add tests
    # Subtest: adding strings
    ok 1 - adding strings
      ---
      duration_ms: 0.262208
      ...
    # Subtest: adding numbers
    ok 2 - adding numbers
      ---
      duration_ms: 0.083291
      ...
    1..2
ok 1 - add tests
  ---
  duration_ms: 1.22125
  ...
1..1
# tests 3
# suites 0
# pass 3
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 0.068583

How often does it reproduce? Is there a required condition?

Any time a child test has no errors.

What is the expected behavior? Why is that the expected behavior?

The data between # Subtest: name and ok 1 - name should be a valid TAP stream, containing at minimum a 1..0 plan.

What do you see instead?

There is no 4-space indented content between the subtest comment and subtest summary test point, meaning that it will likely not be parsed as a subtest, but instead just as a comment and a test point. Which, is fine, and TAP is resilient and flexible like that, but it means that a subtest-aware TAP 13 parser, or a TAP 14 parser, might flag it as unrecognized content, display the comment as diagnostic information, etc., rather that recognize that it was a successful subtest.

A better option would be either to omit the # Subtest comments:

TAP version 13
# Subtest: add tests
    ok 1 - adding strings
      ---
      duration_ms: 0.262208
      ...
    ok 2 - adding numbers
      ---
      duration_ms: 0.083291
      ...
    1..2
ok 1 - add tests
  ---
  duration_ms: 1.22125
  ...
1..1

Or put a TAP stream between the # Subtest comment and summary test point, to make it clear that a subtest block was executed (even though it did not produce any test points).

TAP version 13
# Subtest: add tests
    # Subtest: adding strings
        1..0
    ok 1 - adding strings
      ---
      duration_ms: 0.262208
      ...
    # Subtest: adding numbers
        1..0
    ok 2 - adding numbers
      ---
      duration_ms: 0.083291
      ...
    1..2
ok 1 - add tests
  ---
  duration_ms: 1.22125
  ...
1..1

Additional information

Subtests are not part of TAP 13, so it's not quite accurate to say this is "incorrect", because strictly speaking, it isn't; it's just non-TAP output that a harness can choose to interpret in whatever way they think best. But in practice, among other TAP producers that implement subtests, and which provided the reference examples for TAP 14, they only recognize a subtest as such if it contains an indented TAP stream within the optional introductory comment and the summary test point.

http://testanything.org/tap-version-14-specification.html#subtests

[MoLow]

this was actually added originally to play better with some TAP parser, but I am ok with changing it now that reporters are supported