New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enrich pit CSV with information #122

Closed
monperrus opened this Issue Jun 8, 2017 · 10 comments

Comments

Projects
None yet
3 participants
@monperrus
Copy link
Member

monperrus commented Jun 8, 2017

  • assertion failure or exception error
  • test line of failure/error

PR on PIT?

@oscarlvp

This comment has been minimized.

Copy link
Member

oscarlvp commented Jun 8, 2017

The XML report that comes with PIT provides a bit more of information regarding the mutation, but it doesn't include this kind of runtime information.
A new reporter can be included as a plugin but it will receive the information regarding mutation and coverage already computed. So, in order to do this we have to dig deeper.
Information about a possible thrown exception after running a test (coming from an assertion failure or not) is stored in the TestResult class which is inspected by the Pitest class and passed to a TestListener. A custom listener can be injected when the instance of Pitest is created probably here or modify the one in use and the MutationStatusTestPair class should be also modified to hold the instance of the Throwable so the reporter can access it directly.
With the Throwable at hand we can check the type of the exception thrown and see if it was an assertion failure and we can inspect the stack trace which should contain information about the line of code that failed.

@monperrus

This comment has been minimized.

Copy link
Member Author

monperrus commented Jun 9, 2017

@danglotb

This comment has been minimized.

Copy link
Member

danglotb commented Jun 9, 2017

Another information that could be useful for me is to have the list of all failing test instead of only one.

I see here, pit takes the last failing test.

I will look to open a PR ASAP.

@oscarlvp

This comment has been minimized.

Copy link
Member

oscarlvp commented Jun 12, 2017

@monperrus No, we'll need to change both the code base and also the reporter, but it seems feasible.
@danglotb I did that already. The matrix branch of our pitest fork is a modified pitest that will run all tests related to a mutation instead of exiting with the first test that fails. The behaviour is controlled by a parameter named buildMatrix. Only the command line entry point of pitest has support for this parameter right now, but I think is not hard to include support for maven. I realised now that I haven't pushed the matrix branch into the github repo :)

@monperrus

This comment has been minimized.

Copy link
Member Author

monperrus commented Jun 12, 2017

@oscarlvp

This comment has been minimized.

Copy link
Member

oscarlvp commented Jun 12, 2017

Probably we'll first need to add support for pitest-maven and change all default reporters to include the information of all failed tests. That could take some time. Right now the only reporter supporting this is a custom JSON reporter with I added to descartes.

@monperrus

This comment has been minimized.

Copy link
Member Author

monperrus commented Dec 13, 2017

Right now the only reporter supporting this is a custom JSON reporter with I added to descartes

How to use this custom JSON reporter?

@monperrus

This comment has been minimized.

Copy link
Member Author

monperrus commented Feb 23, 2018

@oscarlvp

How to use the custom JSON reporter?

@oscarlvp

This comment has been minimized.

Copy link
Member

oscarlvp commented Feb 26, 2018

Just add descartes as a dependency for the plugin and later specify JSON as the report file format. like this:

<outputFormats><value>JSON</value></outputFormats>
@monperrus

This comment has been minimized.

Copy link
Member Author

monperrus commented Mar 7, 2018

Thanks.

@monperrus monperrus closed this Mar 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment