New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Maven plugin to check threshold based on pseudo-tested and partially tested methods #22

Open
oscarlvp opened this Issue Sep 29, 2017 · 21 comments

Comments

Projects
None yet
6 participants
@oscarlvp
Collaborator

oscarlvp commented Sep 29, 2017

Create a mechanism to fire a build failure in Maven when the mutation score drops below certain configurable value or the number of strong/weak pseudo tested methods raises above an also configurable threshold.

@vmassol

This comment has been minimized.

Show comment
Hide comment
@vmassol

vmassol Nov 17, 2017

Contributor

Note that this issue was raised as a result of the workshop I did with Oscar on XWiki, see my blog post at http://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes

Contributor

vmassol commented Nov 17, 2017

Note that this issue was raised as a result of the workshop I did with Oscar on XWiki, see my blog post at http://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes

@vmassol

This comment has been minimized.

Show comment
Hide comment
@vmassol

vmassol Jan 29, 2018

Contributor

It would be great to have this issue implemented ASAP so that we can test it on the xwiki project. I believe it's quite easy to do and would allow putting PIT/Descartes in production and get feedback. Thanks!

Contributor

vmassol commented Jan 29, 2018

It would be great to have this issue implemented ASAP so that we can test it on the xwiki project. I believe it's quite easy to do and would allow putting PIT/Descartes in production and get feedback. Thanks!

@oscarlvp

This comment has been minimized.

Show comment
Hide comment
@oscarlvp

oscarlvp Jan 29, 2018

Collaborator

As we discussed briefly last time, this is already supported by PIT and can be used. You may want to check the following PIT configuration parameters:

  • mutationThreshold: An integer specifying the mutation score threshold at which the build must fail.
  • maxSurviving: An integer specifying the maximum number of surviving mutants that a project may allow.

Anyways, I would like to leave this issue open in order to inspect the possibility of implementing a PIT Mojo that fails on the number of pseudo-tested or partially tested methods found, which makes more sense in the context of descartes.

Collaborator

oscarlvp commented Jan 29, 2018

As we discussed briefly last time, this is already supported by PIT and can be used. You may want to check the following PIT configuration parameters:

  • mutationThreshold: An integer specifying the mutation score threshold at which the build must fail.
  • maxSurviving: An integer specifying the maximum number of surviving mutants that a project may allow.

Anyways, I would like to leave this issue open in order to inspect the possibility of implementing a PIT Mojo that fails on the number of pseudo-tested or partially tested methods found, which makes more sense in the context of descartes.

@vmassol

This comment has been minimized.

Show comment
Hide comment
@vmassol

vmassol Jan 29, 2018

Contributor

ok thanks Oscar, I must have forgotten. The strong pseudo-tested criteria makes senses indeed.

BTW, I've tried to use PIT 1.3.1 this morning and I've rebuilt Descartes from sources (0.2-SNAPSHOT - would be nice to have some release BTW) and it failed (some classes/methods were not found). So there's some incompatibility between latest PIT and latest Descartes versions.

Contributor

vmassol commented Jan 29, 2018

ok thanks Oscar, I must have forgotten. The strong pseudo-tested criteria makes senses indeed.

BTW, I've tried to use PIT 1.3.1 this morning and I've rebuilt Descartes from sources (0.2-SNAPSHOT - would be nice to have some release BTW) and it failed (some classes/methods were not found). So there's some incompatibility between latest PIT and latest Descartes versions.

@oscarlvp

This comment has been minimized.

Show comment
Hide comment
@oscarlvp

oscarlvp Mar 20, 2018

Collaborator

Given the current PIT code, we might have to create a new Maven plugin to implement a task that fails on the number of pseudo-tested and partially-tested methods.

Collaborator

oscarlvp commented Mar 20, 2018

Given the current PIT code, we might have to create a new Maven plugin to implement a task that fails on the number of pseudo-tested and partially-tested methods.

@monperrus monperrus changed the title from Threshold check to prevent build in Maven to Implement Maven plugin to check threshold based on #methods Mar 20, 2018

@oscarlvp

This comment has been minimized.

Show comment
Hide comment
@oscarlvp

oscarlvp May 3, 2018

Collaborator

We have a custom reporter in Descartes which is identified as METHODS. This reporter produces a JSON file with a list of pseudo-tested and partially-tested methods. This result can be used to implement the Maven plugin.

Collaborator

oscarlvp commented May 3, 2018

We have a custom reporter in Descartes which is identified as METHODS. This reporter produces a JSON file with a list of pseudo-tested and partially-tested methods. This result can be used to implement the Maven plugin.

@oscarlvp

This comment has been minimized.

Show comment
Hide comment
@oscarlvp

oscarlvp May 3, 2018

Collaborator

Probably the steps are:

  • Implement a new PITest Maven goal, see here. A good approach is to inherit from AbstractPitMojo, in the same way it is done in PitMP.
  • This goal, should use Descartes as the mutation engine, and METHODS as the reporter. See here
  • Inspect the report produced to fail the build if the numbers of pseudo-tested or partially-tested methods go beyond a given threshold.
Collaborator

oscarlvp commented May 3, 2018

Probably the steps are:

  • Implement a new PITest Maven goal, see here. A good approach is to inherit from AbstractPitMojo, in the same way it is done in PitMP.
  • This goal, should use Descartes as the mutation engine, and METHODS as the reporter. See here
  • Inspect the report produced to fail the build if the numbers of pseudo-tested or partially-tested methods go beyond a given threshold.
@spookyvale

This comment has been minimized.

Show comment
Hide comment
@spookyvale

spookyvale May 23, 2018

About the last option, I have a plugin to visualize Descartes reports in Jenkins. You can find the initial version here.
The plugin currently has an option to mark the build as UNSTABLE if the global Mutation Coverage is below a certain treshold (it was a proof of concept), but it can be improved to meet the goal.

spookyvale commented May 23, 2018

About the last option, I have a plugin to visualize Descartes reports in Jenkins. You can find the initial version here.
The plugin currently has an option to mark the build as UNSTABLE if the global Mutation Coverage is below a certain treshold (it was a proof of concept), but it can be improved to meet the goal.

@spookyvale spookyvale closed this May 23, 2018

@spookyvale spookyvale reopened this May 23, 2018

@vmassol

This comment has been minimized.

Show comment
Hide comment
@vmassol

vmassol May 23, 2018

Contributor

Note that the use case of Jenkins is different IMO.

For example in the XWiki case it's not interesting: We need to be able to do this at the level fo the maven build so that it can be executed on developer machines (and since it's in the build it can of course be executed on the CI too, and fail the build too ;)).

Note that we're currently using pitest maven plugin's check mojo.

Contributor

vmassol commented May 23, 2018

Note that the use case of Jenkins is different IMO.

For example in the XWiki case it's not interesting: We need to be able to do this at the level fo the maven build so that it can be executed on developer machines (and since it's in the build it can of course be executed on the CI too, and fail the build too ;)).

Note that we're currently using pitest maven plugin's check mojo.

@nicolabertazzo

This comment has been minimized.

Show comment
Hide comment
@nicolabertazzo

nicolabertazzo May 24, 2018

Contributor

hi @vmassol ,
we are working on the development of maven plugin too.
The jenkins plugin could be useful to display the trend of the metric collected from the mutation testing execution.

Contributor

nicolabertazzo commented May 24, 2018

hi @vmassol ,
we are working on the development of maven plugin too.
The jenkins plugin could be useful to display the trend of the metric collected from the mutation testing execution.

@vmassol

This comment has been minimized.

Show comment
Hide comment
@vmassol

vmassol May 24, 2018

Contributor

I agree that the historical use case (trends) is interesting and not something you can do on the dev machines.

But you don't need the ability to fail the build in the Jenkins plugin for computing trends! ;) (my comment was a reply about the fail-the-CI-job feature).

Also note that it could be interesting to have a Sonar plugin for the trend.

Contributor

vmassol commented May 24, 2018

I agree that the historical use case (trends) is interesting and not something you can do on the dev machines.

But you don't need the ability to fail the build in the Jenkins plugin for computing trends! ;) (my comment was a reply about the fail-the-CI-job feature).

Also note that it could be interesting to have a Sonar plugin for the trend.

nicolabertazzo pushed a commit to nicolabertazzo/pitest that referenced this issue Jun 14, 2018

@nicolabertazzo

This comment has been minimized.

Show comment
Hide comment
@nicolabertazzo

nicolabertazzo Jun 14, 2018

Contributor

I created a fork of pit-test where I added partialyTested and pseudoTested Threshods:
https://github.com/nicolabertazzo/pitest/tree/pseudo-partialy-test
I try to add this features to pitmp but the method to extend (AbstractPitMojo.execute()) is marked as final.

Contributor

nicolabertazzo commented Jun 14, 2018

I created a fork of pit-test where I added partialyTested and pseudoTested Threshods:
https://github.com/nicolabertazzo/pitest/tree/pseudo-partialy-test
I try to add this features to pitmp but the method to extend (AbstractPitMojo.execute()) is marked as final.

@nicolabertazzo

This comment has been minimized.

Show comment
Hide comment
@nicolabertazzo

nicolabertazzo Jun 15, 2018

Contributor

I added two new parameters to AbstractPitMojo of pitest project (i forked it):

And check the thresolds in AbstractPitMojo.execute()

In this way I can compute the pseudo-tested or partially-tested methods for every execution on pitest (with any mutation engine).

The other solution that was suggested from Oscar was to analyze the report produced from decartes methods report engine (METHODS)
This solution works only if you use pit with decartes as mutator engine and methods as report engine.

I tryed to add my solution directly to pitmp-maven-plugin project but I not able to extend AbstractPitMojo.execute() because it is marked as final.

Do you agree with my solution or do you prefer to analyze the report produced from decartes (with methods as report engine).
Do you have any hint for integrate this solution to pitmp-maven-plugin project?

Contributor

nicolabertazzo commented Jun 15, 2018

I added two new parameters to AbstractPitMojo of pitest project (i forked it):

And check the thresolds in AbstractPitMojo.execute()

In this way I can compute the pseudo-tested or partially-tested methods for every execution on pitest (with any mutation engine).

The other solution that was suggested from Oscar was to analyze the report produced from decartes methods report engine (METHODS)
This solution works only if you use pit with decartes as mutator engine and methods as report engine.

I tryed to add my solution directly to pitmp-maven-plugin project but I not able to extend AbstractPitMojo.execute() because it is marked as final.

Do you agree with my solution or do you prefer to analyze the report produced from decartes (with methods as report engine).
Do you have any hint for integrate this solution to pitmp-maven-plugin project?

@vmassol

This comment has been minimized.

Show comment
Hide comment
@vmassol

vmassol Jun 15, 2018

Contributor

Do you have any hint for integrate this solution to pitmp-maven-plugin project?

You already answered it I think since you said:

I try to add this features to pitmp but the method to extend (AbstractPitMojo.execute()) is marked as final.

So this just needs to not be final. I think it's a very bad practice in general to have final methods btw ;) (my personal POV)

Contributor

vmassol commented Jun 15, 2018

Do you have any hint for integrate this solution to pitmp-maven-plugin project?

You already answered it I think since you said:

I try to add this features to pitmp but the method to extend (AbstractPitMojo.execute()) is marked as final.

So this just needs to not be final. I think it's a very bad practice in general to have final methods btw ;) (my personal POV)

@oscarlvp

This comment has been minimized.

Show comment
Hide comment
@oscarlvp

oscarlvp Jun 15, 2018

Collaborator

The concepts of pseudo-tested and partially-tested methods make sense only when using descartes, so there is no problem in using the reporter. The goal to be implemented in the new implementation of AbstractPitMojo should use only descartes as mutation engine.

Collaborator

oscarlvp commented Jun 15, 2018

The concepts of pseudo-tested and partially-tested methods make sense only when using descartes, so there is no problem in using the reporter. The goal to be implemented in the new implementation of AbstractPitMojo should use only descartes as mutation engine.

@CaelInria

This comment has been minimized.

Show comment
Hide comment
@CaelInria

CaelInria Jun 18, 2018

Collaborator

@nicolabertazzo I had the same issue developing PitMP, indeed many methods are private or final in AbstractPitMojo.
So I overwrote shouldRun(), called and the beginning of execute(), and analyse() which actually does the work.

Collaborator

CaelInria commented Jun 18, 2018

@nicolabertazzo I had the same issue developing PitMP, indeed many methods are private or final in AbstractPitMojo.
So I overwrote shouldRun(), called and the beginning of execute(), and analyse() which actually does the work.

@nicolabertazzo

This comment has been minimized.

Show comment
Hide comment
@nicolabertazzo

nicolabertazzo Jun 18, 2018

Contributor

Ok thanks for the hints.

Contributor

nicolabertazzo commented Jun 18, 2018

Ok thanks for the hints.

@nicolabertazzo

This comment has been minimized.

Show comment
Hide comment
@nicolabertazzo

nicolabertazzo Jun 29, 2018

Contributor

To implement Maven plugin to check threshold based on methods i tryed these 3 solutions:

  1. Fork pit project (https://github.com/nicolabertazzo/pitest/commits/pseudo-partialy-test)

    • added two new parameter to pit and checked partialyTested and pseudoTested thresholds on AbstractPitMojo.java
    • store partialyTestedDetected and pseudoTestedDetected on MutationStatistics.java
    • compute partialyTestedDetected and pseudoTestedDetected on MutationStatisticsListener.java
      This solution worked but it's not formally corrected because it coupled pit with descartes
  2. Extend Pitmp (https://github.com/STAMP-project/pitmp-maven-plugin)

    • I tried to extend execute() and analyse() methods of AbstractPitMojo but it's not enought because I need to compute pseudoTested and partialyTested methods (on MutationStatisticsListener) and get the result on analyse method (so extend MutationStatistics.java).
      I dont able to implement the solution in pitmp
  3. Develop a new plugin that check the report created from pit-decartes mutator with METHODS as report engine (similar to the jenkins plugin developed from @spookyvale ).
    The main problem is that i didn't find a way to get the path of the last report produced from decartes. (the solution that i want to investigate is convert the output folder name as date and get the most recent date)

Contributor

nicolabertazzo commented Jun 29, 2018

To implement Maven plugin to check threshold based on methods i tryed these 3 solutions:

  1. Fork pit project (https://github.com/nicolabertazzo/pitest/commits/pseudo-partialy-test)

    • added two new parameter to pit and checked partialyTested and pseudoTested thresholds on AbstractPitMojo.java
    • store partialyTestedDetected and pseudoTestedDetected on MutationStatistics.java
    • compute partialyTestedDetected and pseudoTestedDetected on MutationStatisticsListener.java
      This solution worked but it's not formally corrected because it coupled pit with descartes
  2. Extend Pitmp (https://github.com/STAMP-project/pitmp-maven-plugin)

    • I tried to extend execute() and analyse() methods of AbstractPitMojo but it's not enought because I need to compute pseudoTested and partialyTested methods (on MutationStatisticsListener) and get the result on analyse method (so extend MutationStatistics.java).
      I dont able to implement the solution in pitmp
  3. Develop a new plugin that check the report created from pit-decartes mutator with METHODS as report engine (similar to the jenkins plugin developed from @spookyvale ).
    The main problem is that i didn't find a way to get the path of the last report produced from decartes. (the solution that i want to investigate is convert the output folder name as date and get the most recent date)

@oscarlvp

This comment has been minimized.

Show comment
Hide comment
@oscarlvp

oscarlvp Jul 2, 2018

Collaborator

@nicolabertazzo
You can combine your options 2 and 3. You can implement a Maven Mojo inside PItMP that injects the METHODS output format. This format is a mutation listener in the same way MutationStatisticsListener is. Right now it only produces a JSON file, but we can modify or extend the class to contain properties you can use for the Mojo, or whatever suits your needs, even creating a new listener that makes the build fail. If you want to check the JSON file directly, PIT, by default, creates a folder with the timestamp. So, as you say you can go for the latest report according to the date, and even sorting the folders by name.

Collaborator

oscarlvp commented Jul 2, 2018

@nicolabertazzo
You can combine your options 2 and 3. You can implement a Maven Mojo inside PItMP that injects the METHODS output format. This format is a mutation listener in the same way MutationStatisticsListener is. Right now it only produces a JSON file, but we can modify or extend the class to contain properties you can use for the Mojo, or whatever suits your needs, even creating a new listener that makes the build fail. If you want to check the JSON file directly, PIT, by default, creates a folder with the timestamp. So, as you say you can go for the latest report according to the date, and even sorting the folders by name.

@vmassol

This comment has been minimized.

Show comment
Hide comment
@vmassol

vmassol Jul 13, 2018

Contributor

@nicolabertazzo It would really better if you didn't have to fork pit. Have you opened a ticket on the PIt project to discuss it with them and to see what could be done to make PIT extensible for this use case/needs? This would be the best solution by far.

Note that XWiki is using the PIT Maven plugin and not PIT MP (since it takes way too long to execute on the xwiki codebase). So a solution based on PIT would be good.

Contributor

vmassol commented Jul 13, 2018

@nicolabertazzo It would really better if you didn't have to fork pit. Have you opened a ticket on the PIt project to discuss it with them and to see what could be done to make PIT extensible for this use case/needs? This would be the best solution by far.

Note that XWiki is using the PIT Maven plugin and not PIT MP (since it takes way too long to execute on the xwiki codebase). So a solution based on PIT would be good.

@vmassol vmassol changed the title from Implement Maven plugin to check threshold based on #methods to Implement Maven plugin to check threshold based on pseudo-tested and partially tested methods Jul 13, 2018

@nicolabertazzo

This comment has been minimized.

Show comment
Hide comment
@nicolabertazzo

nicolabertazzo Aug 1, 2018

Contributor

As suggested from @oscarlvp and @CaelInria i extended pitmp-maven-plugin project.
I Created a pull request where:

  • I added two parameters: pseudoTestedThreshold and partiallyTestedThreshold to pitmp mojo
  • I added a MutationResultListenerFactory (CHECKTHRESHOLDS) that fail the build if the number of partialy/pseudo tested methods are above the thresholds
Contributor

nicolabertazzo commented Aug 1, 2018

As suggested from @oscarlvp and @CaelInria i extended pitmp-maven-plugin project.
I Created a pull request where:

  • I added two parameters: pseudoTestedThreshold and partiallyTestedThreshold to pitmp mojo
  • I added a MutationResultListenerFactory (CHECKTHRESHOLDS) that fail the build if the number of partialy/pseudo tested methods are above the thresholds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment