Report mutation test coverage of proposed changes #346

Stephan202 · 2022-11-12T15:52:28Z

Suggested commit message:

Report mutation test coverage of proposed changes (#346)

Thanks to a free Arcmutate OSS license, GitHub Actions now runs Pitest against
files changed in the context of a PR. 

While there, update Pitest's configuration to ignore Refaster rule collection
classes, as mutations of those classes will not impact the associated unit
tests.

See:
- https://www.arcmutate.com
- https://pitest.org

github-actions · 2022-11-12T17:44:40Z

Looks good. No mutations were possible for these changes.
See https://pitest.org/

github-actions · 2022-11-12T17:48:57Z

Surviving mutants in this change: 1
Killed mutants in this change: 1

class	surviving	killed
`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	1	1

See https://pitest.org/

github-actions · 2022-11-12T18:17:29Z

Surviving mutants in this change: 1
Killed mutants in this change: 1

class	surviving	killed
`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	1	1

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed, or the full report under Checks -> Mutation testing.

github-actions · 2022-11-12T18:43:33Z

Surviving mutants in this change: 1
Killed mutants in this change: 1

class	surviving	killed
`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	1	1

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed, or the full report under Checks -> Mutation testing.

github-actions · 2022-11-12T19:09:50Z

Surviving mutants in this change: 2
Killed mutants in this change: 2

class	surviving	killed
`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	2	2

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2022-11-12T19:18:09Z

Surviving mutants in this change: 2
Killed mutants in this change: 2

class	surviving	killed
`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	2	2

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

Stephan202

Before reviewing this PR, consider reading the corresponding blog post.

.github/workflows/pitest-update-pr.yml

pom.xml

rickie

Started reviewing this one, cool stuff!

Added a commit.

This might break things, let's see later when testing the changes 👀.

rickie · 2022-11-18T15:31:36Z

cdg-pitest-licence.txt

@@ -0,0 +1,7 @@
+#Licence file for pitest extensions


We use License everywhere with a "s", let's see if that works here as well.

.github/workflows/pitest-analyze-pr.yml

rickie · 2022-11-19T13:35:07Z

.github/workflows/pitest-analyze-pr.yml

+      - name: Run Pitest
+        run: mvn test pitest:mutationCoverage -DargLine.xmx=2048m -Dverification.skip -Dfeatures="+GIT(from[HEAD~1]), +gitci"
+      - name: Aggregate Pitest reports
+        run: mvn pitest-git:aggregate -DtrailingText="Mutation testing report by [Pitest](https://pitest.org/). Review any surviving mutants by inspecting the line comments under [_Files changed_](${{ github.event.number }}/files)."


Seems fun to me to add some values for killedEmoji and mutantEmoji. Will propose something 👀.

Relevant documentation here.

rickie · 2022-11-19T13:39:11Z

Rebased :).

(Ouch, my bad, I think not rebasing was a deliberate choice, seeing the commits and comments after the commits. Sorry 😞).

Clicking the commits shows the comments on the specific commits, that might help.

Stephan202

I'll let you do the revert and test the other changes 🙃

Stephan202 · 2022-11-22T05:55:57Z

cdg-pitest-license.txt

+#License file for pitest extensions
+#Mon Nov 07 08:41:17 GMT 2022
+expires=07/11/2023
+keyVersion=1
+signature=MhZxMbnO6UovNfllM0JuVWkZyvRT3/G5o/uT0Mm36c7200VpZNVu03gTAGivnl9W5RzvZhfpIHccuQ5ctjQkrqhsFSrl4fyqPqu3y5V2fsHIdFXP/G72EGj6Kay9ndLpaEHalqE0bEwxdnHMzEYq5y3O9vUPv8MhUl57xk+rvBo\=
+packages=tech.picnic.errorprone.*
+type=OSSS


@rickie following your last commit the codes doesn't work anymore; the plugin looks for the exact original file. Now all analysis is skipped. From the logs:

Licenced for evaluation and demonstration purposes only. Expires: 2023-09-20 For use with packages: com.example.*

Stephan202 · 2022-11-22T05:57:13Z

.github/workflows/pitest-analyze-pr.yml

+# uploads the results. The associated PR is subsequently updated by the
+# `pitest-update-pr.yml` workflow. See https://blog.pitest.org/oss-pitest-pr/
+# for details.
+name: "Mutation testing: analysis"


I had that : analysis suffix in an earlier commit, and I removed it on purpose, IIRC. Something to do with how the result is displayed in the GitHub UI. (But would have to revert the last commit to see how/what.)

Suggested change

name: "Mutation testing: analysis"

name: "Mutation testing"

Stephan202 · 2022-11-22T05:59:15Z

cdg-pitest-license.txt

@@ -0,0 +1,7 @@
+#License file for pitest extensions


To be sure: I'd also revert the change to this file. It doesn't follow our standards anyway (missing dot at end of sentence, auto-generated timestamp), so I'd rather keep it exactly as provided.

Or we should replace all comments with something like:

# Arcmutate license for Error Prone Support, requested by sending an email to # support@arcmutate.com.

(Assuming the plugin doesn't care about the comments 👀)

github-actions · 2022-11-24T12:48:44Z

Surviving mutants in this change: 2
Killed mutants in this change: 2

class	surviving	killed
🧟`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	2	2

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2022-11-24T12:49:15Z

Surviving mutants in this change: 2
Killed mutants in this change: 2

class	surviving	killed
🧟`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	2	2

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2022-11-24T13:00:28Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2022-11-24T13:01:08Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2022-11-24T13:14:02Z

Looks good. All 2 mutations in this change were killed.

class	surviving	killed
🎉`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	0	2

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2022-11-24T13:17:09Z

Looks good. All 2 mutations in this change were killed.

class	surviving	killed
🎉`tech.picnic.errorprone.bugpatterns.PrimitiveComparison`	0	2

Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

github-actions · 2022-11-24T16:06:26Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

rickie

Added a commit. Also reverted the test changes.

The setup looks good to me, very cool to have this as part of the build 😄.

Doubting a little about the suggested commit message, we could provide some more information about Arcmutate and the setup we now have. I'm thinking there is a reason that you kept it high-level, so first asking whether there is a particular reason for it 👀 @Stephan202.

pom.xml

rickie · 2022-11-24T16:04:04Z

.github/workflows/pitest-update-pr.yml

+      - name: Set up JDK
+        uses: actions/setup-java@v3.6.0
+        with:
+          java-version: 17


Here we should use the same version as in the existing workflow 17.0.4, right?

github-actions · 2022-11-24T22:03:54Z

Looks good. No mutations were possible for these changes.
Mutation testing report by Pitest. Review any surviving mutants by inspecting the line comments under Files changed.

Stephan202 · 2022-11-26T10:09:16Z

Latest changes LGTM! Also updated the suggested commit message. PTAL :)

rickie · 2022-11-26T10:23:18Z

Nice, looks splendid 😄!

Badbond

Looking good. Nice suggested commit message too. Didn't dive to deep into all the possbile configuration to see if we can improve further. I've got two comments, but will approve nonetheless as they are not blocking this PR for me.

.github/workflows/pitest-analyze-pr.yml

Stephan202 commented Nov 12, 2022

View reviewed changes

.github/workflows/pitest-update-pr.yml Show resolved Hide resolved

pom.xml Show resolved Hide resolved

Stephan202 requested a review from rickie November 12, 2022 19:45

Stephan202 added this to the 0.6.0 milestone Nov 12, 2022

Stephan202 marked this pull request as ready for review November 12, 2022 19:45

rickie added the chore A task not related to code (build, formatting, process, ...) label Nov 18, 2022

rickie reviewed Nov 19, 2022

View reviewed changes

rickie requested a review from Badbond November 19, 2022 13:38

rickie force-pushed the sschroevers/configure-arcmutate branch from db9c60c to 649066f Compare November 19, 2022 13:39

Stephan202 commented Nov 22, 2022

View reviewed changes

rickie force-pushed the sschroevers/configure-arcmutate branch from bfcbc3b to c13fd6d Compare November 24, 2022 12:46

rickie approved these changes Nov 24, 2022

View reviewed changes

Badbond approved these changes Nov 28, 2022

View reviewed changes

.github/workflows/pitest-analyze-pr.yml Show resolved Hide resolved

.github/workflows/pitest-analyze-pr.yml Show resolved Hide resolved

Stephan202 and others added 26 commits November 29, 2022 19:17

Fix uploading

ce11309

Test update-pr

2256a9b

Likely should drop this, too

3850f4d

Permissions

b293b30

Less accurate, but want to see it work

7ad894a

More permissions

4c01656

More...

addbc4f

Doh

221d955

Test

76016b2

What about this?

db5a238

Next step

c12701e

So indeed it isn't inferred

6c64123

Hmm, is the report always generated?

6a3978c

Nicer link?

afb4f43

Final tweaks

6c48f53

Drop test code

0ed96b2

s/Licence/License and suggestions

5e7d7c5

Revert most changes

35e6fe7

Re-introduce test code

26a8b32

Remove bad code to see other emoji

9700f5e

Add properly tested code?

0fc8799

Add missing comment

f8d564c

Specify more specific JDK version for workflows and revert test changes

f6637aa

Drop timestamp comment

3da91ab

Suggestion for linking blogpost on Pitest PR analysis

4054314

Proposal for sharper phrasing

1cc5e8c

Stephan202 force-pushed the sschroevers/configure-arcmutate branch from 7b609a5 to 1cc5e8c Compare November 29, 2022 18:30

rickie changed the title ~~Compute mutation test coverage of proposed changes~~ Report mutation test coverage of proposed changes Nov 30, 2022

rickie merged commit 3303283 into master Nov 30, 2022

rickie deleted the sschroevers/configure-arcmutate branch November 30, 2022 08:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report mutation test coverage of proposed changes #346

Report mutation test coverage of proposed changes #346

Stephan202 commented Nov 12, 2022 •

edited

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

Stephan202 left a comment

rickie left a comment

rickie Nov 18, 2022

rickie Nov 19, 2022

rickie commented Nov 19, 2022 •

edited

Stephan202 left a comment

Stephan202 Nov 22, 2022

Stephan202 Nov 22, 2022

Stephan202 Nov 22, 2022

Stephan202 Nov 22, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

rickie left a comment

rickie Nov 24, 2022

github-actions bot commented Nov 24, 2022

Stephan202 commented Nov 26, 2022

rickie commented Nov 26, 2022

Badbond left a comment

Report mutation test coverage of proposed changes #346

Report mutation test coverage of proposed changes #346

Conversation

Stephan202 commented Nov 12, 2022 • edited

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

github-actions bot commented Nov 12, 2022

Stephan202 left a comment

Choose a reason for hiding this comment

rickie left a comment

Choose a reason for hiding this comment

rickie Nov 18, 2022

Choose a reason for hiding this comment

rickie Nov 19, 2022

Choose a reason for hiding this comment

rickie commented Nov 19, 2022 • edited

Stephan202 left a comment

Choose a reason for hiding this comment

Stephan202 Nov 22, 2022

Choose a reason for hiding this comment

Stephan202 Nov 22, 2022

Choose a reason for hiding this comment

Stephan202 Nov 22, 2022

Choose a reason for hiding this comment

Stephan202 Nov 22, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

github-actions bot commented Nov 24, 2022

rickie left a comment

Choose a reason for hiding this comment

rickie Nov 24, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 24, 2022

Stephan202 commented Nov 26, 2022

rickie commented Nov 26, 2022

Badbond left a comment

Choose a reason for hiding this comment

Stephan202 commented Nov 12, 2022 •

edited

rickie commented Nov 19, 2022 •

edited