notriddle-cerebrum · Jun 16, 2016 · rmg · Sep 7, 2015 · isaacs · Sep 8, 2015
diff --git a/tap-version-14-specification.md b/tap-version-14-specification.md
@@ -0,0 +1,741 @@
+---
+layout: default
+title: TAP 14 specification
+---
+
+# NAME
+
+TAP14 - The Test Anything Protocol v14
+
+## SYNOPSIS
+
+TAP, the Test Anything Protocol, is a simple text-based interface
+between testing modules a test harness. TAP started life as part of
+the test harness for Perl but now has implementations in C/C++,
+Python, PHP, Perl, JavaScript, MatLab, and probably others by the time
+you read this.
+
+This document describes version 14 of TAP.
+
+## THE TAP14 FORMAT
+
+A TAP stream is a UTF-8 encoded text stream representing a set of
+tests.  It should be parsed line by line.  Lines are separated by
+`\n`.
+
+TAP14's general format is:
+
+```
+TAPDocument ->
+  Version Plan Body |
+  Version Body Plan
+```
+
+The Version is always `TAP version 14`.
+
+The Body is a collection of lines representing a test set.
+
+The Plan reports on the number of tests included in the Body.
+
+Any line matching the pattern `/^\s*#.*$/` is treated as a diagnostic
+comment, and MAY be reported to the user or ignored, but MUST be
+ignored for the purposes of interpreting test output semantics.
+
+For example:
+
+```
+TAP version 14
+1..4
+ok 1 Description # Directive
+# Diagnostic
+  ---
+  message: 'Failure message'
+  severity: fail
+  data:
+    found:
+      - 1
+      - 3
+      - 2
+    wanted:
+      - 1
+      - 2
+      - 3
+  ...
+ok 2 Description
+ok 3 Description
+    # Subtest: name of the child test
+    ok 1 this is ok
+    ok 2 this is also ok
+    not ok 3 this is a failure
+      ---
+      message: 'Child Failure'
+      severity: fail
+      ...
+    1..3
+not ok 4 name of the child test
+```
+
+For example, a test file's output might look like:
+
+```
+TAP version 14
+1..4
+ok 1 - Input file opened
+not ok 2 - First line of the input valid
+  ---
+  message: 'First line invalid'
+  severity: fail
+  data:
+    got: 'Flirble'
+    expect: 'Fnible'
+  ...
+ok 3 - Read the rest of the file
+not ok 4 - Summarized correctly # TODO Not written yet
+  ---
+  message: "Can't make summary yet"
+  severity: todo
+  ...
+```
+
+## HARNESS BEHAVIOR
+
+In this document, the "harness" is any program analyzing TAP output.
+
+A harness interpreting a subprocess MUST only read TAP output from
+standard output and not from standard error.
+
+Lines written to standard output matching `/^(not)? ok\b/` must be
+interpreted as test lines. After a test line a block of lines starting
+with `/^ {2}---$/` and ending with `/^ {2}\.\.\.$/` will be interpreted as an
+inline YAML document providing extended diagnostic information about
+the preceding test.
+
+All other lines must not be considered test output.
+
+## TESTS LINES AND THE PLAN
+
+### The Version
+
+To indicate that this is TAP14 the first non-comment, non-whitespace
+line must be:
+
+```
+TAP version 14
+```
+
+A compliant TAP14 Harness SHOULD interpret TAP streams with `TAP
+version 13` according to TAP13 specification.
+
+TAP streams lacking a version number SHOULD be interpreted according to
+the TAP12 specification.
+
+In order to facilitate forward compatibility, a compliant TAP14
+Harness MAY interpret test data according to TAP14, regardless of
+stated version number.  For example, it may accept subtests in TAP12
+or TAP13 streams, even though these were not supported in those
+versions of TAP.
+
+Any stated version number less than 13 is an error.
+
+### The Plan
+
+The Plan tells how many tests will be run, or how many tests have run.
+It's a check that the test file hasn't stopped prematurely.
+
+The Plan MUST appear exactly once, EITHER:
+
+- the first line of TAP output after the Version, or
+- the last line of TAP output prior to the end of the TAP stream.
+
+The Plan matches the regular expression: `/^1\.\.(0|[1-9][0-9]*)( #.*)?$/`.
+
+The Plan specifies how many test points are expected in the test
+stream. For example,
+
+```
+1..10
+```
+
+means you plan on running 10 tests. This is a safeguard in case your
+test file dies silently in the middle of its run.
+
+In certain instances a test file may not know how many test points it
+will ultimately be running. In this case the Plan should be the last
+non-diagnostic line in the output.
+
+A test set missing a Plan MUST be interpreted as a failure by the
+harness.
+
+### The test line
+
+The core of TAP is the test line. A test file prints one test line
+test point executed. There must be at least one test line in TAP
+output. Each test line comprises the following elements:
+
+- ok or not ok
+
+This tells whether the test point passed or failed. It must be at the
+beginning of the line. /^not ok/ indicates a failed test point. /^ok/
+is a successful test point. This is the only mandatory part of the
+line.  Note that unlike the Directives below, ok and not ok are
+case-sensitive.
+
+- Test number
+
+TAP expects the ok or not ok to be followed by a test point number,
+separated by 1 or more space characters from the `/^(not )?ok/`.
+
+For example:
+
+```
+1..3
+ok 1
+not ok 2
+ok 3
+```
+
+If there is no number the harness must maintain its own counter until
+the script supplies test numbers again. So the following test output
+
+```
+1..6
+not ok
+ok
+not ok
+ok
+ok
+```
+
+has five tests. The sixth is missing.  For example, in Perl,
+`Test::Harness` will generate
+
+```
+FAILED tests 1, 3, 6
+Failed 3/6 tests, 50.00% okay
+```
+
+- Description
+
+An optional description may be provided, separated from the ok/not-ok
+and test number by one or more space characters, terminated by either
+a `#` character, or the end of the line.
+
+If the Description starts with a hyphen `-` and any amount of
+whitespace, it SHOULD be discarded by the Harness.  For example, these
+test lines should all be identical:
+
+```
+ok 5 some description
+ok 5 - some description
+ok 5 - some description # and some comment
+```
+
+The harness may do whatever it wants with the description.
+
+- Directive
+
+The test point may include a directive, following a hash on the test
+line.
+
+There are currently two directives allowed: TODO and SKIP. These are
+discussed below.  Any other directive is ignored.
+
+#### Test Line Summary
+
+To summarize:
+
+- ok/not ok (required)
+- Test number (recommended)
+- Description (recommended)
+- Directive (only when necessary)
+
+The regular expression pattern for a test line is:
+
+```
+/^(not )?ok( +[1-9][0-9]*)?( +(- +)[\u0001-\u0009\u000B-\u000C\u000E-\u0022\u0024-\u00FF]+)?( #(.*))?$/
+```
+
+### YAML blocks
+
+If a test line is immediately followed by a 2-space indented block
+beginning with the line `/^ {2}---$/` and ending with the line
+`/^ {2}\.\.\.$/` that block will be interpreted as an inline YAML
+document.
+
+The YAML encodes a data structure that provides more detailed
+information about the preceding failed test.  The YAML document is
+indented with 2 spaces to make it visually distinct from the
+surrounding test results and to make it easier for the parser to
+recover if the trailing `/^ {2}\.\.\.$/` terminator is missing.
+
+For example:
+
+```
+not ok 3 Resolve address
+  ---
+  message: "Failed with error 'hostname peebles.example.com not found'"
+  severity: fail
+  data:
+    got:
+      hostname: 'peebles.example.com'
+      address: ~
+    expected:
+      hostname: 'peebles.example.com'
+      address: '85.193.201.85'
+  ...
+```
+
+The corresponding data structure in JSON would look like this:
+
+```json
+{
+  "message": "Failed with error 'hostname peebles.example.com not found'",
+  "severity": "fail",
+  "data": {
+    "got": {
+      "hostname": "peebles.example.com",
+      "address": null
+    },
+    "expected": {
+      "hostname": "peebles.example.com",
+      "address": "85.193.201.85"
+    }
+  }
+}
+```
+
+If a line occurs within the YAML document that is not indented, or if
+the TAP stream ends before the `/^ {2}\.\.\.$/` terminator line, or if the
+YAML document does not parse as YAML, then it MUST be treated as
+non-compliant data, and handled by the Harness the same as any other
+non-TAP lines.
+
+Currently (2015-10-06) the format of the data structure represented by
+a YAML block has not been standardized.  It is likely that a future
+version of TAP will standardize meanings for some of the fields in the
+YAML diagnostics.
+
+In order to adequately interpret and report the data within a test
+output's YAML block, a full YAML parser is required.  Specifying the
+YAML language is outside the scope of this document.  Harneses MAY
+ignore YAML blocks entirely if the inclusion of a full YAML parser
+would be overly difficult.
+
+## DIRECTIVES
+
+Directives are special notes that follow a # on the test line. Only
+two are currently defined: TODO and SKIP. Note that these two keywords
+are not case-sensitive.
+
+### TODO tests
+
+If the directive starts with `# TODO`, the test is counted as a todo
+test, and the text after TODO is the explanation.
+
+```
+not ok 13 # TODO bend space and time
+```
+
+Note that if the TODO has an explanation it must be separated from
+TODO by a space.  These tests represent a feature to be implemented or
+a bug to be fixed and act as something of an executable "things to do"
+list. They are not expected to succeed. Should a todo test point begin
+succeeding, the harness should report it as a bonus. This indicates
+that whatever you were supposed to do has been done and you should
+promote this to a normal test point.
+
+### Skipping tests
+
+If the directive starts with `# SKIP`, the test is counted as having
+been skipped. If the whole test file succeeds, the count of skipped
+tests is included in the generated output. The harness should report
+the text after # SKIP\S*\s+ as a reason for skipping.
+
+```
+ok 23 # skip Insufficient flogiston pressure.
+```
+
+Similarly, one can include an explanation in a Plan line, emitted if
+the test file is skipped completely:
+
+```
+1..0 # SKIP WWW::Mechanize not installed
+```
+
+## SUBTESTS
+
+A Subtest contains a block of test points grouped together.  This is
+useful for the output of test runners that collect several child
+processes' TAP streams, and output the result in TAP for another
+harness to interpret, or for tests with a predictable number of
+blocks, but where each block might not have a predictable number of
+tests.
+
+A Subtest is a TAP14 stream indented 4 spaces.  It has the following
+format:
+
+- A comment line with the name of the subtest: `/^ {4}# Subtest: (.*)$/`
+- A TAP14 stream, indented 4 spaces.
+- A (non-indented) test point indicating the success or failure of the
+  subtest, with a description matching the Subtest's name.
+
+For example:
+
+```
+TAP version 14
+1..2
+ok 1 - About to do a subtest
+    # Subtest: this is the subtest
+    ok 1 - This is ok
+    ok 2 - I am ok with how things are proceeding
+    not ok 3 - infinite loops # TODO Solve halting problem
+    not ok 4 - this is just a failure
+    1..4
+not ok 2 - this is the subtest
+  ---
+  plan: 4
+  count: 4
+  passing: 2
+  todo: 1
+  failed: 1
+  exitCode: 1
+  ...
+```
+
+If not bailed out, a Subtest MUST be introduced by the indented
+`# Subtest: ...` comment, and MUST be followed with a test point whose
+description matches the Subtest name.  If the Subtest is not
+successful, then the Subtest test point MUST be a `not ok` test point.
+
+A `Bail out!` in a Subtest SHOULD be immediately followed by a
+`Bail out!` in the parent test.
+
+If a Subtest TAP stream does not include a version number, it MUST be
+interpreted as a TAP14 stream anyway.  This is a backwards-compatible
+affordance for pre-TAP14 parsers that complain about seeing a `TAP
+version` line multiple times within a TAP stream.
+
+## OTHER LINES
+
+### Bail out!
+
+As an emergency measure a test script can decide that further tests
+are useless (e.g. missing dependencies) and testing should stop
+immediately. In that case the test script prints the magic words
+
+```
+Bail out!
+```
+
+to standard output. Any message after these words must be displayed by
+the interpreter as the reason why testing must be stopped, as in
+
+```
+Bail out! MySQL is not running.
+```
+
+A Harness SHOULD ignore all lines that come after a bail out.
+
+### Diagnostics
+
+Additional information may be put into the testing output on separate
+lines. Diagnostic lines should begin with a #, which the harness must
+ignore, at least as far as analyzing the test results. The harness is
+free, however, to display the diagnostics.
+
+YAML blocks are intended to replace diagnostics for most purposes but
+consumers should maintain backwards compatibility by supporting them.
+
+### Pragma
+
+A Pragma line indicates that the parser should switch its behavior in
+some way by setting a boolean field.  Pragmas may occur anywhere
+within the TAP stream, similar to Diagnostic comments.
+
+A Pragma starts with the word `pragma`, followed by a space character,
+and a space-separated list of field names, each prefixed with either a
+`+` (true), or a `-` (false).
+
+All Pragmas are false by default.
+
+Pragmas set in Subtests do not affect the state of the pragma field in
+the parent test set.
+
+The only Pragma that is currently specified is `strict`.  Additional
+Pragmas may be added in the future.
+
+To set the strict pragma, use:
+
+```
+pragma +strict
+```
+
+To disable the strict pragma, use:
+
+```
+pragma -strict
+```
+
+- `strict` When set, any non-parsing lines are reported to the user as
+  an error, and cause the test set to be considered a failure.  When
+  disabled, non-parsing lines may be reported to the user in some way,
+  but do not cause the test set to be considered a failure.
+
+### Blank Lines
+
+For the purposes of this specification, a "blank" line is any line
+consisting exclusively of zero or more whitespace characters.
+
+Blank lines within YAML blocks MUST be preserved as part of the YAML
+document, because line breaks have semantic meaning in YAML documents.
+For example, multiline folded scalar values use `\n\n` to denote line
+breaks.
+
+Blank lines outside of YAML blocks MUST be ignored by the Harness.
+
+### Anything else
+
+Any output that is not a version, a plan, a test line, a YAML block, a
+subtest, a diagnostic, a pragma, a blank line, or a bail out is
+incorrect.
+
+When the `pragma +strict` is enabled, incorrect test lines SHOULD
+result in the test set being considered a failure, even if the test
+set is otherwise valid.  When `pragma -strict` is set, incorrect test
+lines MUST NOT result in the test set being considered a failure if
+the test set is otherwise valid.
+
+How or if the incorrect line is displayed to the user by the Harness
+is undefined.
+
+- `Test::Harness` silently ignores incorrect lines, but will become more
+  stringent in the future.
+- `TAP::Harness` reports TAP syntax errors at the end of a test run.
+- `node-tap` prints incorrect lines to standard output, but otherwise
+  ignores them for the purposes of parsing or determining test
+  success.
+
+## EXAMPLES
+
+All names, places, and events depicted in any example are wholly
+fictitious and bear no resemblance to, connection with, or relation to
+any real entity. Any such similarity is purely coincidental,
+unintentional, and unintended.
+
+### Common with explanation
+
+The following TAP listing declares that six tests follow as well as
+provides handy feedback as to what the test is about to do. All six
+tests pass.
+
+```
+TAP version 14
+1..6
+#
+# Create a new Board and Tile, then place
+# the Tile onto the board.
+#
+ok 1 - The object isa Board
+ok 2 - Board size is zero
+ok 3 - The object isa Tile
+ok 4 - Get possible places to put the Tile
+ok 5 - Placing the tile produces no error
+ok 6 - Board size is 1
+```
+
+### Unknown amount and failures
+
+This hypothetical test program ensures that a handful of servers are
+online and network-accessible. Because it retrieves the hypothetical
+servers from a database, it doesn't know exactly how many servers it
+will need to ping. Thus, the test count is declared at the bottom
+after all the test points have run. Also, two of the tests fail. The
+YAML block following each failure gives additional information about
+the failure that may be displayed by the harness.
+
+```
+TAP version 14
+1..2
+    # Subtest: get list of servers
+    1..3
+    ok 1 - connect to database
+    ok 2 - retrieving list of servers
+    ok 3 - list of servers is an array of strings
+ok 1 - get list of servers
+    # Subtest: ping servers
+    ok 1 - pinged diamond
+    ok 2 - pinged ruby
+    not ok 3 - pinged saphire
+      ---
+      message: 'hostname "saphire" unknown'
+      severity: fail
+      ...
+    ok 4 - pinged onyx
+    not ok 5 - pinged quartz
+      ---
+      message: 'timeout'
+      severity: fail
+      ...
+    ok 6 - pinged gold
+    1..6
+not ok 2 - ping servers
+  ---
+  servers_offline:
+    - saphire
+    - quartz
+  ...
+```
+
+### Giving up
+
+This listing reports that a pile of tests are going to be run.
+However, the first test fails, reportedly because a connection to the
+database could not be established. The program decided that continuing
+was pointless and exited.
+
+```
+TAP version 14
+1..573
+not ok 1 - database handle
+Bail out! Couldn't connect to database.
+```
+
+This listing reports a pile of tests where a child test bails out.
+The Harness MAY continue to run subsequent tests, but in most cases,
+SHOULD bubble the bail out up to the parent test.
+
+```
+TAP version 14
+1..54
+    # Subtest: 00-setup.js
+    not ok 1 - database handle
+    Bail out! Couldn't connect to database.
+Bail out! Coulnd't connect to database.
+```
+
+### Skipping a few
+
+The following listing plans on running 5 tests. However, our program
+decided to not run tests 2 thru 5 at all. To properly report this, the
+tests are marked as being skipped.
+
+```
+TAP version 14
+1..5
+ok 1 - approved operating system
+# $^0 is solaris
+ok 2 - # SKIP no /sys directory
+ok 3 - # SKIP no /sys directory
+ok 4 - # SKIP no /sys directory
+ok 5 - # SKIP no /sys directory
+```
+
+### Skipping everything
+
+This listing shows that the entire listing is a skip. No tests were run.
+
+```
+TAP version 14
+1..0 # skip English-to-French translator isn't installed
+```
+
+## Got spare tuits?
+
+The following example reports that four tests are run and the last two
+tests failed. However, because the failing tests are marked as things
+to do later, they are considered successes. Thus, a harness should
+report this entire listing as a success.
+
+```
+TAP version 14
+1..1
+    # Subtest: child program
+    1..4
+    ok 1 - Creating test program
+    ok 2 - Test program runs, no error
+    not ok 3 - infinite loop # TODO halting problem unsolved
+    not ok 4 - infinite loop 2 # TODO halting problem unsolved
+ok 1 - child program
+```
+
+## Creative liberties
+
+This listing shows an alternate output where the test numbers aren't
+provided. The test also reports the state of a ficticious board game
+as a YAML block. Finally, the test count is reported at the end.
+
+```
+TAP version 14
+ok - created Board
+ok
+ok
+ok
+ok
+ok
+ok
+ok
+  ---
+  message: "Board layout"
+  severity: comment
+  dump:
+     board:
+       - '      16G         05C        '
+       - '      G N C       C C G      '
+       - '        G           C  +     '
+       - '10C   01G         03C        '
+       - 'R N G G A G       C C C      '
+       - '  R     G           C  +     '
+       - '      01G   17C   00C        '
+       - '      G A G G N R R N R      '
+       - '        G     R     G        '
+  ...
+ok - board has 7 tiles + starter tile
+1..9
+```
+
+## CHANGES FROM TAP13
+
+- YAML blocks standardized to 2 space indentation
+- Subtests specified to behavior of `Test::More` and `node-tap`.
+- Normative advice regarding exit code for harness programs
+- Examples and usage comments made language-agnostic.
+- Clarification of whitespace and hyphens in test lines.
+- Clarification of handling of incorrect lines.
+- Specification of Pragma lines
+
+## BUGS
+
+Please report issues with this specification to the
+[TestAnything/Specification
+project](https://github.com/TestAnything/Specification).
+
+## AUTHORS
+
+The original TAP documentation (of which this is a hacked about
+version) was written by Andy Lester, based on the original
+`Test::Harness` documentation by Michael Schwern.
+
+TAP13 documentation written by Andy Armstrong.
+
+TAP14 documentation written by Isaac Z. Schlueter.
+
+## ACKNOWLEDGEMENTS
+
+Thanks to Pete Krawczyk, Paul Johnson, Ian Langworth and Nik Clayton
+for help and contributions on this document.  The basis for the TAP
+format was created by Larry Wall in the original test script for Perl
+1. Tim Bunce and Andreas Koenig developed it further with their
+modifications to `Test::Harness`.
+
+As of this writing, the TAP specification is now managed by the
+[TestAnything organization](https://github.com/TestAnything).
+
+## COPYRIGHT
+
+Copyright Michael G Schwern <schwern@pobox.com>, Andy Lester
+<andy@petdance.com>, Andy Armstrong <andy@hexten.net>, Isaac Z.
+Schlueter <i@izs.me>.
+
+This program is free software; you can redistribute it and/or modify
+it under the same terms as Perl itself.  See
+[http://www.perl.com/perl/misc/artistic.html](http://www.perl.com/perl/misc/artistic.html).