New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate pitest tool #3271

Closed
romani opened this Issue Jun 12, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@romani
Member

romani commented Jun 12, 2016

Assigned to @Efremenkov .

Pitest: http://pitest.org/quickstart/maven/

Report: http://checkstyle.sourceforge.net/reports/pitest/
Report built for Commit: 5a0a304
Report generation: 4h (in 4thread, full usage of all cores and memory)

Code change:

05:44 $ git diff
diff --git a/pom.xml b/pom.xml
index 903337a..d20f4df 100644
--- a/pom.xml
+++ b/pom.xml
@@ -989,6 +989,20 @@
         </execution>
       </executions>
     </plugin>
+    <plugin>
+    <groupId>org.pitest</groupId>
+    <artifactId>pitest-maven</artifactId>
+    <version>1.1.10</version>
+    <configuration>
+        <targetClasses>
+            <param>com.puppycrawl.tools.checkstyle*</param>
+        </targetClasses>
+        <targetTests>
+            <param>com.puppycrawl.tools.checkstyle*</param>
+        </targetTests>
+        <threads>4</threads>
+    </configuration>
+</plugin>
     </plugins>
   </build>

command to generate report: mvn test org.pitest:pitest-maven:mutationCoverage

report will be at: firefox target/pit-reports/2016XXXXXXXXX/index.html

TODO:

  • investigate missed point of UTs, fix them.
  • investigate usage in CI to enforce quality of testing in PR

@romani romani added the approved label Jun 12, 2016

@rnveach

This comment has been minimized.

Show comment
Hide comment
@rnveach

rnveach Jun 12, 2016

Member

Interesting tool.

This is called Mutation Testing.

Description:

PIT runs your unit tests against automatically modified versions of your application code. When the application code changes, it should produce different results and cause the unit tests to fail. If a unit test does not fail in this situation, it may indicate an issue with the test suite.
it is actually able to detect whether each statement is meaningfully tested

Description and Examples on how code is modified: http://pitest.org/quickstart/mutators/

Active mutators in CS report:

  • INCREMENTS_MUTATOR
  • VOID_METHOD_CALL_MUTATOR
  • RETURN_VALS_MUTATOR
  • MATH_MUTATOR
  • NEGATE_CONDITIONALS_MUTATOR
  • INVERT_NEGS_MUTATOR
  • CONDITIONALS_BOUNDARY_MUTATOR
Member

rnveach commented Jun 12, 2016

Interesting tool.

This is called Mutation Testing.

Description:

PIT runs your unit tests against automatically modified versions of your application code. When the application code changes, it should produce different results and cause the unit tests to fail. If a unit test does not fail in this situation, it may indicate an issue with the test suite.
it is actually able to detect whether each statement is meaningfully tested

Description and Examples on how code is modified: http://pitest.org/quickstart/mutators/

Active mutators in CS report:

  • INCREMENTS_MUTATOR
  • VOID_METHOD_CALL_MUTATOR
  • RETURN_VALS_MUTATOR
  • MATH_MUTATOR
  • NEGATE_CONDITIONALS_MUTATOR
  • INVERT_NEGS_MUTATOR
  • CONDITIONALS_BOUNDARY_MUTATOR
@rnveach

This comment has been minimized.

Show comment
Hide comment
@rnveach

rnveach Jun 12, 2016

Member

pitest isn't working on Windows with CS (says test suite isn't green but test passes) but it works fine on Ubuntu. I opened an issue with them.

full usage of ... memory

4 gigs of ram + 1 gig swap is not enough, as I kept getting heap errors and than the VM died when all the swap was used.
It finished successfully with 6 gigs of ram and swap filled up almost halfway.

Build Time: [INFO] Total time: 02:43 h (4 cpus/threads)
I did notice alot of timeouts during the process, probably around 400 total.
Other than that, the report looks the same.

Stats by mutation (not in report):

> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 429 Killed 320 (75%)
> org.pitest.mutationtest.engine.gregor.mutators.IncrementsMutator
>> Generated 168 Killed 154 (92%)
> org.pitest.mutationtest.engine.gregor.mutators.VoidMethodCallMutator
>> Generated 5433 Killed 2884 (53%)
> org.pitest.mutationtest.engine.gregor.mutators.InvertNegsMutator
>> Generated 3 Killed 2 (67%)
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 4349 Killed 1828 (42%)
> org.pitest.mutationtest.engine.gregor.mutators.MathMutator
>> Generated 781 Killed 538 (69%)
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 6057 Killed 5239 (86%)
Member

rnveach commented Jun 12, 2016

pitest isn't working on Windows with CS (says test suite isn't green but test passes) but it works fine on Ubuntu. I opened an issue with them.

full usage of ... memory

4 gigs of ram + 1 gig swap is not enough, as I kept getting heap errors and than the VM died when all the swap was used.
It finished successfully with 6 gigs of ram and swap filled up almost halfway.

Build Time: [INFO] Total time: 02:43 h (4 cpus/threads)
I did notice alot of timeouts during the process, probably around 400 total.
Other than that, the report looks the same.

Stats by mutation (not in report):

> org.pitest.mutationtest.engine.gregor.mutators.ConditionalsBoundaryMutator
>> Generated 429 Killed 320 (75%)
> org.pitest.mutationtest.engine.gregor.mutators.IncrementsMutator
>> Generated 168 Killed 154 (92%)
> org.pitest.mutationtest.engine.gregor.mutators.VoidMethodCallMutator
>> Generated 5433 Killed 2884 (53%)
> org.pitest.mutationtest.engine.gregor.mutators.InvertNegsMutator
>> Generated 3 Killed 2 (67%)
> org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
>> Generated 4349 Killed 1828 (42%)
> org.pitest.mutationtest.engine.gregor.mutators.MathMutator
>> Generated 781 Killed 538 (69%)
> org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
>> Generated 6057 Killed 5239 (86%)
@rnveach

This comment has been minimized.

Show comment
Hide comment
@rnveach

rnveach Oct 22, 2016

Member

With the move to JDK 8, pitest wouldn't work for me anymore. See hcoles/pitest#302 .
Temporary solution to get pitest to work is to switch to ECJ (Eclipse's Java Compiler) as the ASM bug is actually a bug in JDKs that they haven't fixed.

My branch: https://github.com/rnveach/checkstyle/commits/pitest
see minor: test and minor: remove resources for ecj

Member

rnveach commented Oct 22, 2016

With the move to JDK 8, pitest wouldn't work for me anymore. See hcoles/pitest#302 .
Temporary solution to get pitest to work is to switch to ECJ (Eclipse's Java Compiler) as the ASM bug is actually a bug in JDKs that they haven't fixed.

My branch: https://github.com/rnveach/checkstyle/commits/pitest
see minor: test and minor: remove resources for ecj

@romani

This comment has been minimized.

Show comment
Hide comment
@romani

romani Jan 10, 2017

Member

All work is done in scope of #3316

Member

romani commented Jan 10, 2017

All work is done in scope of #3316

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment