HTML test report generation

[trungdong]

After an interop test job has finished, a HTML report is generated to give:

• a summary of how many test cases has been passed/failed/skipped, and
• a listing of failed test cases with details about the failure (with a link to the test case repo, if possible)

Please see the nose-htmloutput/nose-html-reporting package for examples.

Given we currently have over 7,500 test cases, the above report will be very useful in helping isolate the failed cases for further investigation, which is challenging with the default text outputs.

[mikej888]

I looked at both nose-htmloutput and nose-html-reporting.

nose-htmloutput:

• https://pypi.python.org/pypi/nose-htmloutput
• https://github.com/ionelmc/nose-htmloutput/
• Both failed and skipped tests are coloured red.
• Ran successfully under Python 2.7.6 and 3.4.0.

$ nosetests -v --with-html --html-file test-report.html prov_interop.interop_tests.test_provpy

nose-html-reporting:

• https://pypi.python.org/pypi/nose-html-reporting
• https://github.com/lysenkoivan/nose-html-reporting/
• A fork of nose-htmloutput.
• Failed and skipped tests are coloured differently.
• Allows browsing of standard output and error for all tests, not just failed ones.
• Ran successfully under Python 2.7.6.

nosetests -v --with-html --html-report test-report.html prov_interop.interop_tests.test_provpy

• Fails under Python 3.4.0 as it's not compliant:

$ nosetests -v --with-html --html-report test-report.html prov_interop.interop_tests.test_provpy
/disk/ssi-dev0/home/mjj/.pyenv/versions/3.4.0/lib/python3.4/site-packages/nose/plugins/manager.py:395: RuntimeWarning: Unable to load plugin html = nose_html_reporting:HtmlReport: No module named 'StringIO'
  RuntimeWarning)
Usage: nosetests [options]

nosetests: error: no such option: --with-html

This is because, from What's New In Python 3.0:

The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.

For running any tests under Python 2.7.6 I'd recommend nose-html-reporting for its richer report format. Unfortunately, as it doesn't work under 3.4.0, that leaves nose-htmloutput as the only option.

Both of these show details about test failures. As for links to the test case repository, one option for this is to provide the repository to the framework as a value in the harness.yaml configuration file. The method:

def test_case(self, index, ext_in, file_ext_in, ext_out, file_ext_out):

in prov_interop/interop_tests/test_converter.py can get this value and print the repository as part of any test failure message.

However, I'm not sure of the value of this, since the test cases repository isn't used by the tests directly, but the local clone of the test cases repository, and the path to this is printed when tests are run anyway.

On which note, the full path to the test case files (and other files) is currently printed. However, as the actual test failure message ("False is not true") was cryptic I've updated (8af65caadfdbf2a14372870f2a38516fc09ce657) this to give a message which includes the names, and paths, of the relevant files e.g.

False is not true : AssertionError: Test failed: 
/disk/ssi-dev0/home/mjj/testcases/test-case1/primer.provx does not 
match out.provx converted from 
/disk/ssi-dev0/home/mjj/testcases/test-case1/primer.json

[trungdong]

Hi @mikej888,

Thank you for looking into this. I wonder if there is an easy way to incorporate the generation of HTML report into the travis jobs. In the case where one sees test failure, they can click on a link and see the report right away.

[mikej888]

There were questions on StackOverflow on how to view test reports for other types of components but the answer was to ensure the test results were printed to the log file. Travis CI does allow build artefacts (which can include test results) to be pushed to Amazon S3 storage, see Uploading Artifacts on Travis CI.

The StackOverflow question Access files stored on Amazon S3 through web browser discusses the next phase, how to make these resources on S3 publicly browsable.

That could offer a possibility to publish the test results in a publicly-visible, and more usable, way.

[trungdong]

Thanks @mikej888, this sounds like a viable option.