Skip to content

Commit

Permalink
docs: slight improvements/formatting /tap-format page
Browse files Browse the repository at this point in the history
  • Loading branch information
@isaacs
isaacs committed Oct 2, 2023
1 parent 844e808 commit 7c3e659
Show file tree
Hide file tree
Showing 2 changed files with 97 additions and 92 deletions.
4 changes: 2 additions & 2 deletions src/docs/_includes/batteries-included.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
### Batteries Included

An easily extensible [<abbr title="Test Anything
Protocol">TAP</abbr>](https://testanything.org) test framework,
with outstanding [TypeScript](./plugins/typescript.md) support, a
Protocol">TAP</abbr>](/tap-format) test framework, with
outstanding [TypeScript](./plugins/typescript.md) support, a
[CLI](/cli.11ty.js), [plenty of assertion
methods](./plugins/asserts.md),
[snapshots](./plugins/snapshot.md), [object/method
Expand Down
185 changes: 95 additions & 90 deletions src/docs/content/tap-format.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,32 +7,33 @@ eleventyNavigation:
---

`tap` is a JavaScript implementation of the [Test Anything
Protocol](http://testanything.org/). TAP is a highly parseable,
human-readable, loosely-specified format for reporting test results.
It rose to popularity in the Perl community, with CPAN's
Protocol](http://testanything.org/). TAP is a highly parseable,
human-readable, loosely-specified format for reporting test
results. It rose to popularity in the Perl community, with CPAN's
[Test](https://metacpan.org/pod/Test::Simple) family.

This protocol is how child test processes communicate their success or
failure with their parent process.
This protocol is how child test processes communicate their
success or failure with their parent process.

Most of the time, you'll view `tap`'s output using one of the
[reporting options](./reporter.md). However, occasionally it's useful
to view the raw output for a variety of reasons. For example, you may
wish to run a test file directly, or store TAP output for later
analysis.
[reporting options](./reporter.md). However, occasionally it's
useful to view the raw output for a variety of reasons. For
example, you may wish to run a test file directly, or store TAP
output for later analysis.

Most of the time, you'll generate TAP by using the functions in
`tap`'s [API](./api.md). But if you have a different kind of program
that you would like to consume with `tap`'s test runner, you can just
print TAP to standard output in any way you please.
The easiest way to generate TAP is by using the functions in
`tap`'s [API](./api.md). But if you have a different kind of
program that you would like to consume with `tap`'s test runner,
you can just print TAP to standard output in any way you please.

This page describes the TAP format that `tap` supports.

## Version

TAP streams generally start with `TAP version 14`. This isn't
strictly required by `tap`, but since some other TAP implementations
_do_ require it, `tap` always outputs `TAP version 14` as the line.
TAP streams generally start with `TAP version 14`. This isn't
strictly required by `tap`, but since some other TAP
implementations _do_ require it, `tap` always outputs `TAP
version 14` as the line.

There's no way to set the version in this library. It's always
14.
Expand All @@ -41,17 +42,18 @@ There's no way to set the version in this library. It's always
14](https://testanything.org/tap-version-14-specification.html).

Since some TAP consumers get upset about an indented version
declaration, the version in Subtest streams is always stripped out.
declaration, the version in Subtest streams is always stripped
out.

## Plan

Every TAP stream must contain a "plan" either at the beginning or the
end of the set of test points. The plan lists the range of test point
IDs that are expected in the TAP stream. It can also optionally
contain a comment prefixed by a `#`.
Every TAP stream must contain a "plan" either at the beginning or
the end of the set of test points. The plan lists the range of
test point IDs that are expected in the TAP stream. It can also
optionally contain a comment prefixed by a `#`.

A plan of `1..0` indicates that the test file is completely skipped,
and no tests are expected.
A plan of `1..0` indicates that the test file is completely
skipped, and no tests are expected.

Examples:

Expand All @@ -61,32 +63,32 @@ Examples:
5..8 # only run tests 5 through 8
```

When consuming a plan, `tap` will accept any of these. However, when
generating test output with `tap`, you may only set the _end_ of the
plan to indicate the number of tests you expect to run (or expect to
have run).
When consuming a plan, `tap` will accept any of these. However,
when generating test output with `tap`, you may only set the
_end_ of the plan to indicate the number of tests you expect to
run (or expect to have run).

Plans cannot be set in the middle of a test set. That is, they have
to come before all the test points, or after all of them.
Plans cannot be set in the middle of a test set. That is, they
have to come before all the test points, or after all of them.

To set a plan in `tap` explicitly, use the `t.plan(n, [comment])`
function. If you end a test by returning a promise or calling
function. If you end a test by returning a promise or calling
`t.end()`, then a plan will be automatically generated at the end
of the stream.

## Test Point

Sometimes called an "assert" or "test line", this is the core of the
TAP format. A test point consists of 4 things:
Sometimes called an "assert" or "test line", this is the core of
the TAP format. A test point consists of 4 things:

1. Either `ok` or `not ok`. This is required. It specifies whether
the test point is a pass or a fail.
2. A optional numeric identifier. This is a check to ensure that test
points are correctly ordered, and that the output is reasonable.
`tap` does not let you set this number explicitly. It assigns test
point IDs based on its own internal counter.
1. Either `ok` or `not ok`. This is required. It specifies
whether the test point is a pass or a fail.
2. A optional numeric identifier. This is a check to ensure that
test points are correctly ordered, and that the output is
reasonable. `tap` does not let you set this number explicitly.
It assigns test point IDs based on its own internal counter.
3. An optional message, which may be prefixed by a `-` character.
4. A directive, prefixed with a `#` character. (See below)
4. A directive, prefixed with a `#` character. (See below)

After a test point, there can be some YAML diagnostics.

Expand All @@ -104,24 +106,25 @@ not ok 2 - that last test point was pretty bare
```

The most common way to generate a test point in `tap` is to use
one of the [assertion methods](./plugins/asserts.md). Test
points are also generated for [subtests](./structure.md), and to
one of the [assertion methods](./plugins/asserts.md). Test points
are also generated for [subtests](./structure.md), and to
communicate failures for unfinished tests, exceeding a plan
count, and anything else that might go wrong.

## Directives

A directive is similar to a comment, but communicates some information
about a test point.
A directive is similar to a comment, but communicates some
information about a test point.

A test point can be marked as `todo` to indicate that it is going to
be implemented later, or `skip` to indicate that the test should not
be performed in the given context. `todo` and `skip` are handled
programmatically to indicate these states. Failures of `todo` or `skip`
test points will not be treated as failure overall.
A test point can be marked as `todo` to indicate that it is going
to be implemented later, or `skip` to indicate that the test
should not be performed in the given context. `todo` and `skip`
are handled programmatically to indicate these states. Failures
of `todo` or `skip` test points will not be treated as failure
overall.

A test point associated with a Subtest can also have a `# time=...`
directive indicating how long the subtest took to run.
A test point associated with a Subtest can also have a `#
time=...` directive indicating how long the subtest took to run.

Example:

Expand All @@ -135,25 +138,25 @@ ok 3 - child test # time=12ms
1..3
```

In this case, we see a test that failed, but that's expected, because
it hasn't been implemented yet. Then, test point #2, we're skipping
because we're on Windows. Lastly, there's a child test stream that
took 12ms to complete.
In this case, we see a test that failed, but that's expected,
because it hasn't been implemented yet. Then, test point #2,
we're skipping because we're on Windows. Lastly, there's a child
test stream that took 12ms to complete.

Overall, a passing test stream :)

To set a `todo` or `skip` directive, pass `{ todo: reason }` or
`{skip: reason}` in either an assert or subtest method. If you don't
wish to provide a reason, you can pass `{todo: true}` or `{skip:
true}`. You can also mark subtests as `todo` by omitting the callback
function.
`{skip: reason}` in either an assert or subtest method. If you
don't wish to provide a reason, you can pass `{todo: true}` or
`{skip: true}`. You can also mark subtests as `todo` by omitting
the callback function.

## YAML Diagnostics

Diagnostics can be used to provide additional details about a test
point. They are a YAML object indented by 2 spaces, starting with
`---` and ending with `...`. YAML diagnostics are associated with the
preceeding test point.
Diagnostics can be used to provide additional details about a
test point. They are a YAML object indented by 2 spaces, starting
with `---` and ending with `...`. YAML diagnostics are associated
with the preceeding test point.

```tap
TAP version 13
Expand All @@ -180,23 +183,25 @@ not ok 2 - failing, gonna tell you why
1..2
```

In `tap`, diagnostics are printed by default with failing tests, but
not with passing tests. You can change this default by setting
`TAP_DIAG=0` in the environment to make it not print diagnostics with
failing tests or by setting `TAP_DIAG=1` to make it print diagnostics
with passing tests by default. Setting `{ diagnostic: true }` in a
test point options object will always print diagnostics. Setting `{
diagnostic: false }` will always omit diagnostics.
In `tap`, diagnostics are printed by default with failing tests,
but not with passing tests. You can change this default by
setting `TAP_DIAG=0` in the environment to make it not print
diagnostics with failing tests or by setting `TAP_DIAG=1` to make
it print diagnostics with passing tests by default. Setting `{
diagnostic: true }` in a test point options object will always
print diagnostics. Setting `{ diagnostic: false }` will always
omit diagnostics.

## Subtests

A [subtest](./structure.md) is an indented TAP stream that is a
child of the current set of tests. It can be used to group test
child of the current set of tests. It can be used to group test
points together, consume the output of a TAP-producing child
process, or run tests asynchronously.

The summary test point ensures that TAP consumers that ignore indented
lines will at least report on the passing status based on the summary.
The summary test point ensures that TAP consumers that ignore
indented lines will at least report on the passing status based
on the summary.

```tap
1..2
Expand All @@ -212,15 +217,15 @@ ok 1 - child test
ok 2
```

The most common way to run subtests is via `t.test(...)`. See
The most common way to run subtests is via `t.test(...)`. See
[Subtests](./structure.md) for more info.

## Pragma

Pragmas are a way to set arbitrary switches on the parser.

The only switch that is treated specially is `strict`. When in strict
mode, any non-TAP data is treated as an error.
The only switch that is treated specially is `strict`. When in
strict mode, any non-TAP data is treated as an error.

```tap
TAP version 13
Expand All @@ -235,20 +240,20 @@ ok 3 - because garbage data is allowed in non-strict mode
```

Set pragms in `tap` by doing `t.pragma({ keys: values, ... })`.
The object can contain any number of keys, but only `strict` has any
meaning to `tap` itself.
The object can contain any number of keys, but only `strict` has
any meaning to `tap` itself.

## Bail out!

Sometimes a set of tests hits a state where there's no point
continuing. Or, perhaps you just wish to stop on the first failure to
work on errors one at a time with a faster turnover.
continuing. Or, perhaps you just wish to stop on the first
failure to work on errors one at a time with a faster turnover.

In this case, TAP allows a "bail out". A bail out is much more
extreme than a test point failure. It means that everything should
come to a halt, all the way up to the highest level test harness.
Nothing should come after a bailout. Any plan is dropped, test points
ignored, and so on.
In this case, TAP allows a "bail out". A bail out is much more
extreme than a test point failure. It means that everything
should come to a halt, all the way up to the highest level test
harness. Nothing should come after a bailout. Any plan is
dropped, test points ignored, and so on.

```tap
TAP version 13
Expand All @@ -273,19 +278,19 @@ Anything that starts with a `#` and is not a directive or subtest
prefix is treated as a comment, and ignored.

Anything that isn't parseable as one of the above types of lines
is considered "extra" non-TAP data. In strict mode, extra output
is an error. In non-strict mode, it's ignored.
is considered "extra" non-TAP data. In strict mode, extra output
is an error. In non-strict mode, it's ignored.

The `tap` runner ignores comments unless `--comments` is
provided, in which case it is printed to stderr. Non-TAP data is
provided, in which case it is printed to stderr. Non-TAP data is
passed through the reporting engine and printed to the top-level
process standard output. This means that `console.log('foo')`
process standard output. This means that `console.log('foo')`
will make its way to the top level, instead of being swallowed by
a reporter.

You can generate comments by doing `t.comment('foo')`. This
You can generate comments by doing `t.comment('foo')`. This
function takes any arguments that can be passed to
`console.log()`. For example, `t.comment('number %d and\nobj =',
`console.log()`. For example, `t.comment('number %d and\nobj =',
1, { foo: 'bar' })` would output:

```tap
Expand Down

0 comments on commit 7c3e659

Please sign in to comment.