Add support for generating an html report for test results

[valeraz]

I'd like to enable the ability to generate an html report when executing python tests with pants v2. The pytest-html plugin solves this problem. Per discussion on Slack, pants' sandbox currently prevents input/output from this plugin to enter/exit the sandbox and we could, potentially, solve this generically by capturing everything in under a plugin-output directory.

[benjyw]

@valeraz this is implemented in #11741, which will be in the next dev release (2.4.0.dev3).

The catch is, you have to tell pytest-html to output under extra-output/, which is the dir we hoist out of the sandbox and copy under dist/.

So for example:

./pants test <target> -- --html=extra-output/report.html 

Will place the report(s) under dist/.

[stuhood]

@valeraz this is implemented in #11741, which will be in the next dev release (2.4.0.dev3).

We can backport to 2.3.x if need be as well.

[valeraz]

Wow, that was quick! @stuhood backporting to 2.3.x would be awesome!

[Eric-Arellano]

We can backport to 2.3.x if need be as well.

Now that 2.3.0 stable is released, we should think twice about backporting a new feature imo. We generally only backport bug fixes. Granted, this is lower risk than other new features.

Fwit, now that 2.3.0 is stable, we can start release candidates for 2.4.0 soon.

[benjyw]

We released 2.4.0.dev0 on Feb 27, so 2.4.0rc0 is probably due in a week from today.

@valeraz how urgent is this for your team?

[valeraz]

Not super-urgent. We can wait until the next RC release.

[MartykQ]

Is there any built-in feature that would allow merging all html files generated by pytest-html into one report?

[benjyw]

No, presumably we'd want to merge XML reports and then generate one HTML from one XML. So the question is if pytest-html can do stuff like that.