[Experimental] Initial attempt at adding JUnit XML support

[jmdacruz]

Related to #49. Most CI tools include some support for interpreting JUnit XML output, extracting information about total executed tests vs. passed, capturing stdout/stderr from test execution, etc. This is an attempt at implementing some very rudimentary support for JUnit XML in mutmut.

In order to get the report in this new format, simply run mutmut junitxml (similar to running mutmut results)

Notes for discussion:

• This could be exposed as a modified/parameter to the results command, instead of a new command (it's just a different way of reporting the results).
• What to show on the stdout and stderr attributes. Right now I'm simply adding Mutant.line.line, but ideally we want to show the diff of the mutant. It seems the code for generating the diff is on the __main__.py file, so maybe that can be refactored and reused there?
• What is a failure vs. error vs skipped? I'm using failure for surviving mutants, error for timeouts, and skipped for both untested and suspicious. A flag could be added to determine if suspicious mutants should be skipped or considered failures/errors

[codecov-io]

Codecov Report

Merging #55 into master will increase coverage by 0.86%.
The diff coverage is 79.54%.

@@            Coverage Diff             @@
##           master      #55      +/-   ##
==========================================
+ Coverage   82.44%   83.31%   +0.86%     
==========================================
  Files           3        3              
  Lines         735      767      +32     
==========================================
+ Hits          606      639      +33     
+ Misses        129      128       -1

Impacted Files	Coverage Δ
mutmut/cache.py	83.64% <78.37%> (-1.19%)	⬇️
mutmut/__main__.py	73.33% <85.71%> (+2.37%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 192de1e...d5d55ad. Read the comment docs.

[jmdacruz]

I don't think we need the groupby here, unless we want to create separate test suites.

[jmdacruz]

Actually we do, so that mutants are listed on the report by file (even if they are not really grouped)

[jmdacruz]

This is what the rendered JUnit XML output looks like with a tool such as xunit-viewer (get it here). Install it and run it like xunit-viewer --port=8080 --results=output.xml, where output.xml is the result of doing mutmut junitxml > output.xml:

And this is what a suspicious mutant looks like (see that the unified diff of the mutant is included on the report):

[jmdacruz]

Refactoring the code that was performing the unified diff on __main__.py, and bringing it here so that we can also use it for the JUnit report

[boxed]

I like it! Fairly small diff and seems pretty feature complete. The new command seems reasonable to me. As for the status of the suspicious mutants, I'm fine with having them just not be in the output.

This change seems pretty ready to merge as is I think. What do you think?

[jmdacruz]

Glad you liked it! Let me add an argument to the command in order to manipulate what to do with the suspicious mutants, and then we can merge, sounds good?

[boxed]

Great!

[jmdacruz]

I added a couple of parameters to change the policy for dealing with suspicious and untested mutants. A few notes for future reference:

• I don't think I would be able to maintain the patch coverage target without changing the way the tests are parameterized, since the execution of the foo.py test does not generate suspicious mutants. The only alternative would be to just test the junit report function by mocking the Mutant DAO object. Let me know if you want me to do that now.
• I added some documentation on the README.rst file. I noticed that there is improper use of backticks on the document (those work on markdown files, but not on RST), so I used them too on purpose so that this can fix in a separate PR (either use proper RST or move to markdown altogether)
• I added the arguments for the report as global options in click, but ideally we should add support for click's subcommand feature: http://click.palletsprojects.com/en/7.x/commands/. This provides greater control on arguments for each individual command. I'll open a separate issue to discuss this (the change would be backwards compatible)

That's it, ready to merge (I suggest "squash and merge", since there is a lot of noise on this PR)!

[boxed]

Yea the codecov thing is bs and should be ignored. I should google on how to turn that off sometime :P

I use RST for the documentation generation with Sphinx. You could have fixed it in this PR that's fine. I'm not a Puritan for tiny stuff like that.