Editing Everything you need to know about software testing report formats

TestAnythingProtocol (TAP)

The Test Anything Protocol (TAP) is a protocol to allow communication between unit tests and a test harness. It allows individual tests to communicate test results to the testing harness in a language-agnostic way. Originally developed for unit testing of the Perl interpreter in 1987, producers and parsers are now available for many development platforms.

TAP is widely supported as a unit testing framework by test harness and automated test framework, in almost all languages. There is a specification.

Sample of output in TestAnythingProtocol format:

TAP 13
1..4
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
not ok 4 - Summarized correctly # TODO Not written yet

SubUnit

SubUnit is a streaming protocol for test results, which allows communication between unit tests and a test harness. It was originally developed for unit testing in 2005 by Robert Collins. There is a specification. Format has two versions: v1 (plain text) and v2 (binary).

Sample of output in SubUnit v1 format:

progress: 28704
time: 2016-07-05 12:17:02.290433Z
test: bzrlib.doc.api.DocFileTest(/usr/lib64/python2.7/site-packages/bzrlib/doc/api/branch.txt)
time: 2016-07-05 12:17:02.314892Z
successful: bzrlib.tests.blackbox.test_add.TestAdd.test_add_from_subdir(view-aware) [ multipart
]
skip: bzrlib.tests.test_http.TestBadProtocolServer.test_http_has(pycurl,HTTP/1.1) [
pycurl doesn't check the protocol version
]

JUnit

xUnit is the collective name for several unit testing frameworks that derive their structure and functionality from Smalltalk's SUnit. SUnit, designed by Kent Beck in 1998, was written in a highly structured object-oriented style, which lent easily to contemporary languages such as Java and C#. Following its introduction in Smalltalk the framework was ported to Java by Kent Beck and Erich Gamma and gained wide popularity, eventually gaining ground in the majority of programming languages in current use. The names of many of these frameworks are a variation on "SUnit", usually replacing the "S" with the first letter (or letters) in the name of their intended language ("JUnit" for Java, "RUnit" for R etc.). These frameworks and their common architecture are collectively known as "xUnit".

Specification:

• PHPUnit
• NUnit
• xUnit.net v1 XML Format
• xUnit.net v2 XML Format
• https://llg.cubic.org/docs/junit/
• https://github.com/windyroad/JUnit-Schema
• https://github.com/junit-team/junit5/blob/master/platform-tests/src/test/resources/jenkins-junit.xsd
• http://help.catchsoftware.com/display/ET/JUnit+Format
• https://github.com/apache/maven-surefire/blob/master/maven-surefire-plugin/src/site/resources/xsd/surefire-test-report.xsd
• https://xunit.github.io/docs/format-xml-v2.html
• https://github.com/bluebird75/luaunit/tree/master/junitxml

Sample of output in JUnit format:

<testsuites>
<testsuite errors="1" failures="0" name="" tests="4" time="55.051">
    <testcase classname="sudoku_app.tests.test_sudoku.TestMainWindow" name="test_settings_tab(with mouse)" time="9.320">
        <error type="testtools.testresult.real._StringException">
        </error>
    </testcase>
    <testcase classname="sudoku_app.tests.test_sudoku.TestMainWindow" name="test_best_scores_tab(with mouse)" time="4.459"/>
    <testcase classname="sudoku_app.tests.test_sudoku.TestMainWindow" name="test_enter_and_cancel(with mouse)" time="7.384"/>
    <testcase classname="sudoku_app.tests.test_sudoku.TestMainWindow" name="test_enter_and_clear_number(with mouse)" time="8.104"/>
 </testsuite>
</testsuites>

Comparison Table

	TAP	SubUnit v1	SubUnit v2	JUnit
Human-readable	Yes	Yes	No	No
Language-agnostic	Yes	Yes	Yes	Yes
Year	1988	2006	2008
Test Suites Support	No	Yes?	Yes?	Yes
Extensibility	Yes, with embedded YAML	No	No	No
Specification	Yes	Yes	Yes	No
Time support	Yes, with embedded YAML	Yes	Yes	Yes
File Attachments	Yes, Base64 in embedded YAML	Yes, Base64	Yes	No
Consumers	pytap13, shouldertap	Tribunal, testrepository, subunit2sql, junitxml2subunit, os-testr, generate-subunit, subunit2html, subunit-trace		Wikipedia: List of unit testing frameworks
Producers	TAP Producers, pytest	go-subunit, subunit, pytest, NodeJS, Rust, nose, Wikipedia: List of unit testing frameworks, pytest-subunit		Wikipedia: List of unit testing frameworks, pytest
Syntax Highlighting	highlight.js, Pygments, Vim, Emacs, Rouge	highlight.js	No	highlight.js, Vim, Rouge
Users	MariaDB, PostreSQL, Perl, NginX, GIT SCM, CRIU, LibVirt etc	OpenStack, Bazaar, Ubuntu	OpenStack, Samba etc	GlusterFS, Mercurial etc

Test Report Systems

• Badger JUnit
• Allure (Jenkins Plugin) JUnit
• Tapper TAP
• Test Results Analyzer Plugin (Jenkins Plugin) TAP JUnit
• app--prove--history TAP
• TAP-DOM-Db TAP
• Smolder TAP
• testres TAP JUnit SubUnit
• TestCube JUnit