New functionality (combine reports etc)

[JamesMTSloan]

I have added some new functionality to HtmlTestRunner that you will hopefully find useful.

• reports can be combined by setting the HTMLTestRunner combine_reports kwarg to True: this will produce a single file containing a separate table for each TestCase. (combining results to one html report #6)
• the output argument is now optional and defaults to a folder called reports under the current working directory ("./reports/") (HtmlTestRunner cannot be directly used with Django DiscoverRunner #12 I think)
• the user can pass extra variables to templates using the template_args kwarg in the form of a dict (see readme for an example of its use)
• filename timestamps are optional using the add_timestamp kwarg
• any stdout from a test method will be available in the expanded view for that test method in a report (View passed testcases and show log messages in html report? #27 I think)
• traceback details are also available in the report (Enhancement - Failure and Error Tracebacks #28)
• template formatting has changed
• readme has been expanded and some of the wording changed
• option added to have reports automatically open in new browser tab using the open_in_browser kwarg
• TestCases with underscores in name are supported (Module Names with Underscores #29)

EDIT: More features have been added as I've gone (see below). This started too big and has only gotten bigger so sorry about that.

[oldani]

HI @JamesMTSloan,

Thanks for this huge PR, i will be reviewving and testing it.

[durisv]

Hi

I have one idea of functionality. It may be good to add your own report name.

[JamesMTSloan]

Gracias @oldani, let me know.

@durisv: that is something that I have considered yeah and it is a good idea. At the moment you can set the title of the reports (i.e. the text at the top of the page) but I expect you are referring to the filename (i.e. what goes before the timestamp and '.html').

For combined reports or test suites with only one test this would be straight forward but would be complicated for cases where separate reports are being produced because they could all end up with the same name. One solution could be to always combine reports and remove the option of not doing so.

Alternatively, we could append the TestCase name to then end of whatever you say should be the title when making separate reports. Do you have an example of what you would like to be able to achieve easily?

[JamesMTSloan]

I've quickly added optional report naming with the last strategy outlined above. Use the report_name kwarg.

For example if you wanted a combined report to be produced with the name "MyReports.html" you could do so as follows:

import unittest
from HtmlTestRunner import HTMLTestRunner

unittest.main(
    testRunner=HTMLTestRunner(
        combine_reports=True,
        add_timestamp=False,
        report_name="MyReports"
    )
)

I'll add it to the README when I can.

[PBatomic]

Is there any recent activity on this? I would like to use JamesMTSloans changes, as combining report is huge thing for me now.

Anyway great job on HtmlTestRunner, really making things good! 👍

[JamesMTSloan]

I've added a couple of psuedo-tests just to make demonstrating the changes a little easier for @oldani. Really it is just down to whenever he gets the time though.

[Eki23]

Hi, I'm also very interested in this (and very new to python, unittest and selenium, too).

Is there any way I can try this new feature before it it included in the trunk?

[JamesMTSloan]

@Eki23 you could always check out from my own master branch and install using the -e flag for pip install.
Alternatively, if you just want to have a play around you could just download the couple of files that this project comprises and add them to whatever workspace you are using, importing them into your scripts straight from there.

[Eki23]

Hi @JamesMTSloan

Thanks for the suggestion.

I tried your branch, and I managed to make the aggregate work

I see that the tests within a testcase are not sorted alphabetically (as it is with this module) but rather grouped by pass/fail/warning/skip status, which is also not working for our case were tests within a test case are executed sequentially and we are expecting the reports to reflect that order too.

Since I am still in the prototype stage, I will stick with the current implementation, without aggregate. If past that stage the requirements for the reporting insist on the aggregate, I might get back to your master branch and try to implement a custom solution.

Thanks for your input. I've definitely learned a handful of things.

[JamesMTSloan]

If that is the case then my implementation needs to be changed. I think I can see where the grouping is being effected and will look at changing it when I get the chance. Thanks for the feedback.

[JamesMTSloan]

OK @Eki23 I've pushed a fix for the alphabetical ordering. Cheers.

[Eki23]

That's greate @JamesMTSloan !!!

It's working great for me now!

I might get back to you for some recommendations on how best use this.

Thanks a lot.

[JamesMTSloan]

Any updates @oldani ? If there's anything you'd like me to do to make reviewing and testing easier then let me know.

[kimfucious]

Hola @JamesMTSloan,

I love the idea of combining test case results in a single report!

I checked out your master branch and installed v1.1.2 locally; however, I'm having an issue:

import unittest
import HtmlTestRunner
from tests.test_anagrams import AnagramsTests
from tests.test_bst import BinarySearchTreeTests
from tests.test_capitalize import CapitalizeTests


loader = unittest.TestLoader()
anagrams = loader.loadTestsFromTestCase(AnagramsTests)
bst = loader.loadTestsFromTestCase(BinarySearchTreeTests)
capitalize = loader.loadTestsFromTestCase(CapitalizeTests)

suite = unittest.TestSuite([anagrams, bst, capitalize])

runner = HtmlTestRunner.HTMLTestRunner(combine_reports=True)
runner.run(suite)

Returns

TypeError: __init__() got an unexpected keyword argument 'combine_reports'

There doesn't seem to be a way to raise an issue on your repo, so I'm asking here 😊

[JamesMTSloan]

Hola @kimfucious

My initial guess would be that you have the original version of HtmlTestRunner installed in your virtual environment instead of this branch and that your import statement is pulling in the version without the new report combining functionality.

If you debug the code without the combine_reports kwarg you can see what file it takes you to.
It is clear to me that the combine_reports option exists here.

If the above isn't taking you to the expected file then you may need to uninstall the version of HtmlTestRunner without this kwarg, and reinstall from the branch. Look into virtual environments in Python if you are not already familiar with them as they are useful (necessary) for managing these kinds of package management issues.

Cheers.

[kimfucious]

Spot on, Señor @JamesMTSloan!

My globals were a mess. I wound up discovering pipenv, and it's a beautiful thing.

You know what else is beautiful? The test results from your master branch. ¡Olé! 🎉

Looking forward to this PR getting merged.

[hannu40k]

Hello,

Hopefully this PR gets merged soon. Super useful.

[JamesMTSloan]

Support added for sub-tests in sub-tables (#35)
Support added for skipped test skip reasons (#36)

I've also tweaked the layout a bit. This required some hacking away at the template and with HTML and I'm no ace with HTML so if anyone wants to suggest some improvements that'd be nice. Either edit the template directly or edit a generated report and give it to me to change the template to reflect the changes.

Note that summaries do not include individual subtests.

[oldani]

Hi there,

@JamesMTSloan sorry it's this long time and I haven't merged this PR yet. Please add your name to the AUTHORS file. I will merge this temporarily in a feature branch so everybody can install it via pip + git url, until I can take time review it, merge it with some changes I have locally and release v2 which will have a lots improvements. Thanks for all the work done here, next time try not to put a lots of thing in a single PR (different features/issues on same PR). With the foloowing cmd you will be able to install this branch:

pip install git+https://github.com/oldani/HtmlTestRunner@feature/combined_report

[misterk72]

Hello,
first thank you for this great feature. That was exactly what I was looking for.
I tried it on my test cases and ran into 2 problems:

• if you pass only one test case to the test runner the HTML generation fails with this error : AttributeError: 'list' object has no attribute 'copy'.
• I got a "File name too long" error because all the test suites names are concatenated to generate the html file and I have a lot of test suites...
  I patched these two problems here: https://github.com/misterk72/HtmlTestRunner

[williamfzc]

Good feature but 6 months passed...
Hopefully this PR gets merged soon.

[kyuubi326]

Hello,
thank you, this feature is exactly what I needed, the support for the subtests is great!
But I really miss the original functionality which showed the description instead of the test names in the table (as is shown in the readme).
Is there maybe some way to fix that?

Otherwise: well done!
Greetings

[oldani]

Sorry everyone I have abandoned this project, I will be merging this and releasing a new version.

@JamesMTSloan Thanks again for this.

[oldani]

Version 1.2 has been released with all these wonderful changes:)

Sorry everybody

[ramasa2]

That's greate @JamesMTSloan !!!

It's working great for me now!

I might get back to you for some recommendations on how best use this.

Thanks a lot.

Hi @Eki23 ,

I know i am replying to very old post .I am expecting a similar html report which clubs all the test suite results into one single html report .But i am unable to get that even after setting combine_reports=True. HTMLTestRunner is at version 1.2.1 .

My code below:
runner = HtmlTestRunner.HTMLTestRunner(template='template.html', output='./', combine_reports=True,report_name="MyReports", add_timestamp=False, failfast=kwargs['fail_fast'])

#Here i have a for loop to iterate over each test suite containing multiple test steps(test cases)
test_suite = unittest.TestSuite()
test_suite.addTest(objTC) #objTC contains my testcases
try:
res = runner.run(test_suite)

This is generating a single html report ,for example there are 3 test suites : test suite 1 generates a html report MyReports.html the same gets overwritten by test suite2 and then test suite3 . I want all test suite1, test suite2 and test suite3 results combined in single report (MyReports.html) .

Can someone please help me i am totally stuck.

[JamesMTSloan]

@ramasa2 it isn't possible to tell from the edited code sample that you have posted but there is a chance that your for loop isn't constructed correctly. If test_suite is being created again every iteration of the loop then you will not get the results you want.

If that is indeed the case then the solution would be to create test_suite first, then start your for loop that adds the test cases. Then, outside the loop once its completed, call runner.run().

# create the test suite
test_suite = unittest.TestSuite()

# add the tests one by one
for objTC in my_tests:
    test_suite.addTest(objTC) #objTC contains my testcases

# run the tests all together and generate the report
res = runner.run(test_suite)

[ramasa2]

@JamesMTSloan Thanks for your swift response .You are right my suite is being created in every iteration .Because each test suite is different set of test cases .The suggested solution above combines all the test cases (cant differentiate which test case belongs to which test suite ).

Please refer Eki23's July 13 2018's post .The screenshot shows Jmeter Test, Report and logs Test and Baseclass Test .I wonder how did it happen to generate such report . I want such report .Am i missing to provide any info here ?

Each of the test suites have their own test steps separately .

[ramasa2]

Can runner.run take list of test suites i,e runner.run([suite1,suite2,suite3 ..]) where suite1 has 8 test cases ,suite2 has 4 tcs and so on .

[ramasa2]

Can some one please help me on this . I am unable to get solution for this .

[Eki23]

Hi @ramasa2

This is what I have:

For each test suite
	...
	suite = unittest.TestSuite()
	
	For each test in the test suite
		...
		suite.addTest(...)
		...
	
	...
	runner = HtmlTestRunner.HTMLTestRunner(stream=runner_output, combine_reports=True, ...)
	runner.run(suite)

This is generating one report for each test suite.
Each report will have a series of test cases results (unittest.TestCase instance).
Each test case result has multiple lines, one for each testMethod inside the TestCase.

I do not combine the test suites reports in a single one, but if I wanted to do this, I would move the instantiation of the test suite before the loops, would iterate through all test suites and test cases to add them to the suite instantiated, and have the runner instantiated and run after, outside of the nested loops.

Hope this helps.