This plug-in adds support to TAP test result files to Jenkins. It lets you specify an ant-like pattern for a directory that contains your TAP files and scans and creates views for your test results in Jenkins.
TAP Plug-in depends on tap4j - a TAP implementation for Java, and on the Jenkins JUnit Plug-in.
NOTE: You may get errors if the JUnit Plug-in is not active in your Jenkins instance (see JENKINS-27227 for more).
The plug-in looks for TAP files like the following one.
1..2
ok 1 - Yahoo!
not ok 2 - org.tap4j.Error...
When a TAP stream like the above is found, the plug-in delegates the parsing the tap4j. The results of tap4j parsing are then analysed, organized and displayed to the user as graphs and custom pages.
You can test your TAP streams with tap4j, using InstantTAP.
One of the easiest ways to learn something new is by examples. So here we will show some examples of TAP streams. For being human-friendly, it shouldn't confuse you. We will use comments to explain each line.
1..2 # a plan stating that we have two tests cases, from 1 to 2.
ok 1 - Yahoo! # the first test result was executed successfully and has a description of ' - Yahoo!'.
not ok 2 - org.tap4j.Error... # unfortunately, the second test result failed. The description here was used to display some nasty Exception.
The plug-in cannot handle prove's default output (since it includes more information than simply TAP, and causes tap4j parser to fail). The best way to handle the prove output is by using Perl module TAP::Harness::Archive. Supposing you have your tests under t/ directory, you can create another directory (say, output) and archive your TAP tests with prove by using a command line similar to the below:
prove t/ --archive outputThe result files will be stored under output/t/. You can use a pattern in the plug-in configuration like t/*/.t.
The following is a TAP with attachments, using YAMLish. If you are familiar with YAML, this example should be very easy to read.
1..2
ok 1
not ok 2 - br.eti.kinoshita.selenium.TestListVeterinarians#testGoogle
---
extensions:
Files:
my_message.txt:
File-Title: my_message.txt
File-Description: Sample message
File-Size: 31
File-Name: message.txt
File-Content: TuNvIGNvbnRhdmFtIGNvbSBtaW5oYSBhc3T6Y2lhIQ==
File-Type: image/png
...The Files entry has an array of files. You have to Base64 encode your content data.
Subtests let you group several TAP streams unto a single one. This way, you can organize your tests in similar fashion to JUnit or TestNG test suites. Indentation is important for TAP subtests.
Subtests and YAMLish are not officially in TAP 13 specification
1..3
ok 1 - First test
1..2
ok 1 - This is a subtest
ok 2 - So is this
1..2
ok 1 - This is a subtest
ok 2 - So is this
ok 2 - An example subtest
ok 3 - Third test- Install the Jenkins TAP Plug-in using the Plug-in Manager or manually by copying the
.hpi(or.jpi) file - Check the option to publish TAP, configure a pattern (and other settings)
- Execute your build and analyze the results
- English (American)
- Portuguese (Brazil)) Work in Progress
- Spanish (Thanks to César Fernandes de Almeida) Work in Progress
- If the file type of the TAP report is considered as binary by the Jenkins webserver then the TAP plugin does not
consider this file for inclusion in TAP reports (see #15813
for further details). To make sure your TAP report is considered for inclusion use e.g. the file name suffix
.tap( so instead of a file namedreportusereport.tap).
{width="300"}For commercial support, please get contact us via @tupilabs
- This plug-in is going to be part of "Make your tests speak TAP" presentation in JCertif by Bruno P. Kinoshita, in September 2011. In this presentation will also be presented the tap4j project, how to enable TAP in JUnit and TestNG, integrate Perl and Java tests and an Eclipse TAP editor.
- Gábor Szabó (2009), Test Reporting system: Smolder wish-list http://szabgab.com/blog/2009/07/test-reporting-system-smolder-wish-list.html.
- Gábor Szabó (2009), Reporting Test Results http://szabgab.com/blog/2009/04/reporting-test-results.html.
- tap4j - The TAP implementation for Java.
- Test Anything Protocol (official webpage).
- Performance tests with phantomjs and yslow (uses the plug-in for plotting the TAP results)
- TAP Plugin for Matlab
JCertif - Make your tests speak TAP -
Speaker: Bruno P. Kinoshita
September, 2011 - Brazzaville, Congo
Moved to CHANGES.md.
- Update Jenkins API
- Update tap4j to match the latest protocol specification (13 and 14)
- Remove deprecated code
- Simplify code (it's old!)
- Add configurations like validate number of tests with test plan, a TODO causes a test to fail
- Diagnostics image gallery Done! Fixed as a new plug-in
- Diagnostics exception code formatting
- Add a link to open the file in the Build workspace (think about remote and local issues) Done!
The idea of the plug-in surged after tap4j was created. After learning about Smolder, it became evident that Jenkins could be used as a replacement for it. All that was needed was just adding TAP support to Jenkins and implementing a nice UI to display the test results. After some messages in jenkins-dev-list, Max and Nick commented about their need to show test results in a different manner than how Jenkins was doing at that moment. Soon after that Max, Nick, Bruno (tap4j) and Cesar (tap4j) started to work together, exchanging mail messages and discussing an initial design for this plug-in.
In July 2011 the first version of the plug-in was ready to be released. The graph code used here was adapted from TestNG Plugin (big thanks to the development team, great work). The diagnostic (YAMLish) was implemented in Jelly + Java + CSS. And the road map was incremented based on what Gábor Szabó posted about Smolder and testing reports in his blog (see resources for links).
You can use a heredoc to write a TAP file with Shell, and use it with the plug-in. This is useful for testing.
#!/bin/bash
for x in {1..100}; do
cat > $x.tap <<EOF
ok 1
not ok 2
ok 3 # SKIP
not ok 4
ok 5
ok 6
ok 7
EOF
done ;