Jenkins plugin to show unit-test, tesng, jtreg and JCK reports summaries, diffs and details
Plugin is build as mvn clean install -Phpi, otherwise Diff cli tool jar with dependecies is built
The plugin reads archived gzipped xml files prdoduced by junit/testng/jtreg/jck suites (or anyhow else generated) runs.
- Implementation details
- Job run details
- Project details
- View
- Denylist and Allowlist
- Project Settings
- View Settings
- Limitations
- Diff cli tool
- Future work
- For developers
The xml reports you recieve, should be post processed a bit, and compressed. Zip, tar.gz and tar.xz (including xml.gz and xml.xz) are supported. Compressed, as plugin is used with reports hundrets of megabytes large. And postprocessed as various engines generates various reports. Thus the xml files should be gathered into archives, which are later considered as suites:
The level of granularity is up to you. The tar.gz archvies are later cached as two relatively small json files - one with listing for diff, one with stack traces of failures. Latest cache is the properties file with total/run/passed/failed/error/skipp keys to be reused via https://github.com/judovana/jenkins-report-generic-chart-column (and used for quicker renderig of graphs of this plugin itself)
Quick overview in build page:
simple, but a lot of saying JTREG Reports Total: X, Error: Y, Failed: Z message is printed, where JTREG Reports is link to dretails page:
Here yo can see several items:
- dummy navigation which allwos you to skip back and fwd. It do not check status of run, so it can return 404 on failed run
- table, with suites. Each suite have detailed total/passed/failed/error skip details. Table is followed by
- listing of faiures. The listing is clear, and contains failed tests only. Each failure contains expandable
- trace. The listing is followd by
- diff. Diff is done first for added/removed suites. For unchanged suites, the diff of suite itself is done. You can immediately see exact fixed and freshly failing tests. After the diff, you can see
- full listing of testsuite. The ful listing is stripped to 1000 lines due to performance. The output is generated by the rutines of Diff cli tool, so please apologise a bit of cryptic "passed or missing" and "faile dor error" output. To see the full listing, you must use the cli.
- on start of each section is cross roada allowing direct jumps to table, list of issues, diff and full listing
On the project page, you can see several graphs:
- number of failures shows how much test failed and how much had error. The graph is scalled, but in usual world, there is very small number of failures, and similarly small number of errors, so scale is not affected.
- number of total tests shows how much test exists in suite and how much wasrun. The graph is scalled, but in usual world, there is very small number of skipped tests so both lines are of same value and so scale again should not be affected.
- regression chart shows how much test had changed status. Green bar - number of tests which get fixed. Red bar, number of tests which started to fail. This graph is here to prevent overlook of X fixed tests and (same) X of freshly failing tests. IN such case, your total number of passes/failures is constant, and thus invisible on normla charts.
Each chart have detailed tooltip and is capable of click which takes you directly to details page:
You can place jtreg charts also to jenkins view, so you can eyball all your testruns in one gaze.
Here the jtreg plugin is the left most graph. The two right most graphs are charts from properties
You could have noted, that the graphs are scaled. Sometimes it happens that fails 100x more tests then usually. This is killing the scale, and you can miss the regression. Such a build deserves to be denylisted once the issue is solved. On contrary, allowlist is here to allow you to comapre just some selected runs. Both lists are space separated list of regexes against job name (usually #NUMBER or some_custom_name). The lists are shared betwen project and view.
Project settings are simple - you set glob expression for files to be considered as archives with results, and select how many tests you wish to show
View settigns are fully inherited from project settings. So the only thing you do in view is to set the order - column for a chart
The imitations are clear from shared settings with all pros and cons and quite clumsy comparsion of exact jobs (w/b lists) and impossible comparsion between jobs.
To workaround limitations, and add possibility to post-process results, the hpi and jars contains main class of io.jenkins.plugins.report.jtreg.main.diff.CompareBuilds whcih allows (based on the director with yor jobs) to comapre or list practically anything. The launcher can look like:
set -x
set -e
# we need to get jenkins on classpath...
unpackDir=\$(dirname \$(mktemp))/$jenkinsWar
if [ ! -d \$unpackDir ] ; then
mkdir \$unpackDir
unzip $jenkinsSrc -d \$unpackDir
fi
set +x
jars=\`find \$unpackDir | grep \\.jar \`
CP=""
for jar in \$jars ; do
CP=\$CP\":\"\$jar
done
CP=\$CP\":\"$jenkins_main_dir/jenkins-report-jck-jar-with-dependencies.jar
set -x
/opt/jdk/bin/java -cp \$CP -Djenkins_home=$jenkins_main_home io.jenkins.plugins.report.jtreg.main.diff.CompareBuilds $@
It can spwn plain tex, colored tex, or even html, so it i easy to be deployed as service (there is a wrapper for this too - Service ) together with jenkins.
options DIR1 DIR2 DIR3 ... DIRn
or
options JOB_NAME-1 buildPointer1.1 buildPointer1.2 ... jobPointer1.N JOB_NAME-2 buildPointer2.1 buildPointer2.2 ... jobPointer2.N ... JOB_NAME-N ...jobPointerN.N
If you use only jobname, then its builds will be listed:
options:
-output=color|html|html2
default output is 'plain text'. 0-1 of -output is allowed.
-view=all-tests|diff|diff-details|diff-list|diff-summary|diff-summary-suites|hide-misses|hide-negatives|hide-positives|hide-totals|info|info-hide-details|info-problems|info-summary|info-summary-suites
default view is 'all'. 0-N of -view is allowed. Best view is -view=diff-list -view=info -view=info-hide-details
job pointers are numbers. If zero or negative, then it is 0 for last one, -1 for one beofre last ...
Unknow job will lead to listing of jobs.
When using even number of build pointers, you can use -fill switch to consider them as rows
Another strange argument is -keep-failed which will include failed/aborted/not-existing builds/dirs during listing.
The cli works with absolute and relative job IDs, and strictly cooperates with stdout/err (so consider 2>/dev/null sometimes)
- with no param it prints name
- with no job it prints jobs
- with no build it lists builds (of given job)
On everything else it do the listings and/or comparsion (or fails). Eg summary:
diff:
html view:
web-cli - Service - is nothing more then wrapper around io.jenkins.plugins.report.jtreg.main.diff.CompareBuilds and is doing nothing more then resending stdout/err to browser request!! There is hardcoded port of 9090 in Sevice class.
Html output is much more clumsy, but the listing of switches and jobs is live, and also ajax is helping here a bit. Also yu can send results as URL, so it have its cases
- To figure out how to make job-name based view on top of the shared settings.
- make cli more user friendly
This plugin depends on https://github.com/judovana/jenkins-chartjs-plugin
In the compile phase of the report-jtreg-plugin module a script.sh/.bat is run. The script copies the target directory of the already compiled jtreg-report-lib module into the target directory of the plugin module.
This is done because of the Jenkins security hardening (https://www.jenkins.io/blog/2018/03/15/jep-200-lts/). The plugin can't load classes from external modules, and it throws a class filter exception.
The plugin should work correctly with this "workaround", however, if it doesn't, run Jenkins with this switch, it should solve the problem: -Dhudson.remoting.ClassFilter=io.jenkins.plugins.report.jtreg.model.Report,io.jenkins.plugins.report.jtreg.model.ReportFull,io.jenkins.plugins.report.jtreg.model.Suite,io.jenkins.plugins.report.jtreg.model.Test,io.jenkins.plugins.report.jtreg.model.TestOutput
This plugin is multimodule lib, plugin and two external services. Thus the autorelease do not work as expected.
To release, you have to cd report-jtreg submodule which contans report-jtreg-plugin artifactId and here run manually mvn release:preapre and mvn release:perform. During preapare, always adjsut all modules, not jsut dependent ones (choice of 0).
Sometimes, you may need to fake the build bit, and to change project version to not-snapshot one (the upcoming release), and mvn clean install lib, so it is in local m2 repos. then rewert the change, and proceed as described above.
After release, you may need to fix your poms, eg:
diff --git a/pom.xml b/pom.xml
index ccca4f6..3cb2495 100644
--- a/pom.xml
+++ b/pom.xml
@@ -25,7 +25,7 @@
</modules>
<properties>
- <revision>2.6</revision>
+ <revision>2.7</revision>
<changelist>-SNAPSHOT</changelist>
<gitHubRepo>jenkinsci/report-jtreg</gitHubRepo>
<chartjs.version>1.0.2.6</chartjs.version>
diff --git a/report-jtreg/pom.xml b/report-jtreg/pom.xml
index f357a9a..7eea371 100644
--- a/report-jtreg/pom.xml
+++ b/report-jtreg/pom.xml
@@ -37,7 +37,7 @@
<dependency>
<groupId>io.jenkins.plugins</groupId>
<artifactId>report-jtreg-lib</artifactId>
- <version>2.6</version>
+ <version>${revision}${changelist}</version>
<scope>provided</scope>
</dependency>
</dependencies>
Because vhere mvn release:prepare and release:perform can remove the variables properly, it can not restore them as expected: https://github.com/jenkinsci/report-jtreg-plugin/commit/fe6d56b43304a41c85ec1d4eea965d257891e8cf -> https://github.com/jenkinsci/report-jtreg-plugin/commit/5bebd159ef746604dcddfbb250517cddbf6723f5 (amended)