TAP Y J Specification
Clone this wiki locally
TAP-Y and TAP-J are test streams. They are essentially the same except for the underlying format used, which are YAML and JSON repsectively.
The following overview will use the YAML format. Becuase the YAML format is a plan format, not using an special YAML tags, it is easy to convert to JSON to get the equivalent TAP-J format. TAP-J documents follow all the same feild rules as TAP-Y, but are represented as a stream of JSON documents, one per line, instead of YAML documents.
This document describes Revision 4 of the specification.
TAP-Y is a plain YAML stream format. Only YAML core types are used: scalar, sequence and mapping.
A YAML stream is composed a sequence of YAML documents, each divided by
a start document marker (
---). Each document MUST have a
field which designates it a
document MAY have an
extra entry which contains an open mapping for
suite document marks the beginning of a forthcoming stream of tests,
i.e. a test suite. All TAP-Y streams MUST begin with a suite
--- type: suite start: 2011-10-10 12:12:32 count: 2 seed: 32154 rev: 4
start field marks the date and time testing began. It MUST be
an ISO-8601 formated timestamp.
count field indicates the total number of test units forethcoming.
If the number of test units is unknown, the total can be omitted or
~ (nil). The total should only indicate the number of
test units, not any enumeration of test cases.
seed is provided if the test runner has randomized the order of
execution unit tests. The seed can be used to reproduce the same order.
rev field provides the version of the TAP-Y/J format that is
being used. The specification will change little, if at all, as it
become more mainstream. But just in case, having a revision field
ensures things will work if they do change by allowing consuming
apps to adjust to any future variation.
case type indicates the start of a test case.
--- type: case subtype: feature label: Multiplication level: 0
The case document MAY provide a
subtype which is a label for the
type of test case. For example, a test framwework that uses Gherkin
nomenclature would classify a test case as a "feature".
The case document SHOULD provide a
label that is a free-form string
describing the nature of the test case.
level field is used to notate sub-case heiararchies. By default the
value is assumed to be
0, which means the case is not a subcase. If
it indicates that the case is a subcase of the previous zero-level
case, and so on for higher levels. Subcases should proceed sequentially.
If a case contains both tests and subcases, the tests must come first in the
test type indicates a test procedure. A unit MUST have a
one of five possible values:
Unit documents vary somewhat based on the status. But they all share
Here is an example of a passing unit document.
--- type: test subtype: step status: pass setup: foo instance label: multiples of two expected: 2 returned: 2 file: test/test_foo.rb line: 45 source: ok 1, 2 snippet: - 44: ok 0,0 - 45: ok 1,2 - 46: ok 2,4 coverage: file: lib/foo.rb line: 11..13 code: Foo#* stdout: '' stderr: '' time: 0.01
status, all test documents MUST have a
A test document MAY provide a
setup field, which is used to describe
the setup for the unit test.
Tests SHOULD also give an
returned value, if relavent
to the nature of the test. For example, the most common test assertion is
expected would be
returned would be
4. Although desirable this can be a difficult piece
of information for some test frameworks to provide, so it is the most often
A test SHOULD also have a
line number for source file location.
This is the location of the test definition itself.
A test SHOULD provide the line of
source code for the test.
This will be the line of code that
line number references.
snippet lines, the source line should be stripped of whitespace.
snippet is like
source but provides surronding context. It MAY be
a verbatim string, in which case it MUST have an odd number of lines with
the source line in the center. Or, it MAY be an ordered map of verbatim
- line: source. Using an ordered map the line numbers may start
and end wherever, but they MUST be consecutive and the source line MUST
be among them.
coverage subsection MAY be provided, in which can have three optional
file specifies the source file being
targeted by the test,
line specifies the line number, range of line
1..4) or an array of such, and
code specifices the
language construct being targeted. For example, for Ruby
code might be
if the test targets the
bar method of the
stderr fields contain any output produced while running
time is the number of seconds that have elapsed since the
the suite start time.
If a test has a status other than
pass it MUST also provide a
subsection which is used to describe the nature of the failure, error or
--- type: test subtype: step status: fail label: multiples of two setup: foo instance expected: 2 returned: 1 file: test/test_foo.rb line: 45 source: ok 1, 2 snippet: - 44: ok 0,0 - 45: ok 1,2 - 46: ok 2,4 coverage: file: lib/foo.rb line: 11..13 code: Foo#* exception: message: | (assertion fail) must_equal 1 2 file: test/test_foo.rb line: 50 source: 1.must_equal == v snippet: - 49: v = 2 - 50: 1.must_equal == v - 51: '' backtrace: - test/test_foo.rb:50 - test/test_foo.rb:45 time: 0.02
exception section MUST give the
message, describing the nature
of the failure or exception. In this subsection,
the location in code that triggered the exception or failed assertion.
Like the originating test code, a
source and code
snippet SHOULD also
It MAY also provide a system
Q. Why supply a code snippet when the file and line are already given. Can't a test reporter just look up the code itself? A. Of course it can, but if the TAP-Y document is being consumed remotely it might not have easy access the file being tested. While this may be of rare use it none the less provides the TAP-Y consumer some view of the code with having to do additional processing.
note type is used to interject a message between tests that
is not tied to a specific unit or case. It has only a few fields.
--- type: note text: This is an example note.
The note document is simply used to interject any information the tester might want to know, but doesn't properly fit elsewhere in the stream. A note cna appear any where in the document stream prior to the tally.
Final & Tally
tally types are the same. The difference is only that a
entry is a running tally, and can technically occur anywhere in the document
final entry on the other hand incidates the end of a test suite,
which will be followed by an end-document-marker (
--- type : final time : 0.03 counts: total: 2 pass : 1 fail : 1 error: 0 omit : 0 todo : 0 ...
A tally/final document MUST provide a counts mapping with the
total number of
tests (this MUST be same as
count in the suite document if it was given)
and the totals for each test status. It SHOULD also give the time elapsed
since the suite time.
Tally documents are very rare, if used at all. They only make sense for very
large test suites as a progress report mechanism. As a rule of thumb, TAP-Y/J
consumer apps will ignore them unless a configuration option (e.g.
As mentioned, the test stream ends when a full ellipsis (
As you can see TAP-Y streams provides a great deal of detail. They are not intended for the end-user, but rather to pipe to a consuming app to process into a human readable form.
Glossery of Fields
count field provides the total number of tests being executed. It SHOULD
be given in the header, if possible, and it MUST be given in the footer.
Additional data, not specifucally designated by this sepecification can
placed within an
extra section of any document without worry that future
versions of the specification will come into conflict with the field name.
The field MUST be a mapping. The key namespace is a free-for-all, so use it
with that in mind.
file field provides the name of the file in which the test is defined,
or where th test failed/errored.
line field provides the line number of the file on which the
definition of the test begins, or is the line number of where the
A subsection used to sepcify the nature of a non-passing test.
For tests without a
pass status, the message provides the explination
for the failure or error. Usually this is just the error message produced by
the underlying exception. The
pass type can have the message field too,
but it will generally be ignored by TAP consumers.
snippet field is either a verbatim string or an ordered mapping of line
number mapped to the source code for that line. While
source it also contains extra lines of code before and after the
line for context.
snippet is a string it MUST consist an odd number of lines, the same
number before and after the source line in the center, unless the line occurs
at the begining or the end of the file. The number of lines before and after is
arbitrary and up to the producer, but should be the same on either side. Three
to five is generally enough.
source field is a verbatim copy of the source code that defines the test.
This may be just the first line of the definition. In classic TAP this
The suite decument provides date/time information for when a suite of tests
began being tests. The filed MUSTbe in ISO standard format
status field designates the status of a test document. Valid values
In comparison to the classic TAP format,
pass is equivalent to
error are akin to
not ok, where
fail is "not ok" with regards
to a test assertion and
error is "not ok" becuase of a raised coding error.
Tests with an
omit status do not need to be provided in the document stream,
so this status might not appear often in practice. But if a producer chooses to
do so this status simply means the test is purposefully being disregarded for
some reason. The
exception subsection is used to clarify that reason.
On the other hand,
todo means the test will be used in the future
but implementation has not been completed. It serves as reminder to developers
to write a missing test.
The footer MUST provide counts for all status categories and the total.
This is like
count in the suite entry but broken down into status groups.
The tests and the footer SHOULD have the
time elapsed since starting the
tests given in number of seconds.
Each document MUST have a type. Valid types are
suite type can only occur once, at the start of the stream. All other
types may occur repeatedly in between, although the
tally type will
generally only occur at the end of a stream.
case type marks the start of a testcase. All
documents following it are considered a part of the case until a new case
document occurs with the same level.