Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enrich pit CSV with information #122

Closed
monperrus opened this issue Jun 8, 2017 · 10 comments
Closed

enrich pit CSV with information #122

monperrus opened this issue Jun 8, 2017 · 10 comments

Comments

@monperrus
Copy link
Member

@monperrus monperrus commented Jun 8, 2017

  • assertion failure or exception error
  • test line of failure/error

PR on PIT?

@oscarlvp
Copy link
Member

@oscarlvp oscarlvp commented Jun 8, 2017

The XML report that comes with PIT provides a bit more of information regarding the mutation, but it doesn't include this kind of runtime information.
A new reporter can be included as a plugin but it will receive the information regarding mutation and coverage already computed. So, in order to do this we have to dig deeper.
Information about a possible thrown exception after running a test (coming from an assertion failure or not) is stored in the TestResult class which is inspected by the Pitest class and passed to a TestListener. A custom listener can be injected when the instance of Pitest is created probably here or modify the one in use and the MutationStatusTestPair class should be also modified to hold the instance of the Throwable so the reporter can access it directly.
With the Throwable at hand we can check the type of the exception thrown and see if it was an assertion failure and we can inspect the stack trace which should contain information about the line of code that failed.

@monperrus
Copy link
Member Author

@monperrus monperrus commented Jun 9, 2017

@danglotb
Copy link
Member

@danglotb danglotb commented Jun 9, 2017

Another information that could be useful for me is to have the list of all failing test instead of only one.

I see here, pit takes the last failing test.

I will look to open a PR ASAP.

@oscarlvp
Copy link
Member

@oscarlvp oscarlvp commented Jun 12, 2017

@monperrus No, we'll need to change both the code base and also the reporter, but it seems feasible.
@danglotb I did that already. The matrix branch of our pitest fork is a modified pitest that will run all tests related to a mutation instead of exiting with the first test that fails. The behaviour is controlled by a parameter named buildMatrix. Only the command line entry point of pitest has support for this parameter right now, but I think is not hard to include support for maven. I realised now that I haven't pushed the matrix branch into the github repo :)

@monperrus
Copy link
Member Author

@monperrus monperrus commented Jun 12, 2017

@oscarlvp
Copy link
Member

@oscarlvp oscarlvp commented Jun 12, 2017

Probably we'll first need to add support for pitest-maven and change all default reporters to include the information of all failed tests. That could take some time. Right now the only reporter supporting this is a custom JSON reporter with I added to descartes.

@monperrus
Copy link
Member Author

@monperrus monperrus commented Dec 13, 2017

Right now the only reporter supporting this is a custom JSON reporter with I added to descartes

How to use this custom JSON reporter?

@monperrus
Copy link
Member Author

@monperrus monperrus commented Feb 23, 2018

@oscarlvp

How to use the custom JSON reporter?

@oscarlvp
Copy link
Member

@oscarlvp oscarlvp commented Feb 26, 2018

Just add descartes as a dependency for the plugin and later specify JSON as the report file format. like this:

<outputFormats><value>JSON</value></outputFormats>

@monperrus
Copy link
Member Author

@monperrus monperrus commented Mar 7, 2018

Thanks.

@monperrus monperrus closed this Mar 7, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants