report aggregation process implemented for multi-project #243

MikeSafonov · 2020-11-28T13:50:15Z

Greetings
I have implemented a report aggregation process using pitest-aggregator

szpak

Thanks for your PR!

See the separate remarks.

szpak · 2020-11-28T14:49:09Z

build.gradle

@@ -34,6 +34,7 @@ sourceSets {

 dependencies {
    implementation localGroovy()
+    implementation "org.pitest:pitest-aggregator:1.5.2"


It is somehow problematic. PIT allows to dynamically change used PIT version and it might be not compatible with agregator's 1.5.2. I have to think if (and how) it could be made up.

I have some idea, but please make the functional tests green to allow me to test it.

...y/info/solidsoft/gradle/pitest/functional/MultiprojectReportAggregationFunctionalSpec.groovy

src/main/groovy/info/solidsoft/gradle/pitest/report/PitestReportAggregator.groovy

...y/info/solidsoft/gradle/pitest/functional/MultiprojectReportAggregationFunctionalSpec.groovy

src/main/groovy/info/solidsoft/gradle/pitest/report/PitestReportAggregator.groovy

src/main/groovy/info/solidsoft/gradle/pitest/PitestPlugin.groovy

src/test/groovy/info/solidsoft/gradle/pitest/PitestAggregatorPluginTest.groovy

.../solidsoft/gradle/pitest/functional/AcceptanceTestsInSeparateSubprojectFunctionalSpec.groovy

MikeSafonov · 2020-12-11T11:49:27Z

Hello. Any updates?

szpak · 2020-12-12T01:59:47Z

Hello. Any updates?

Unfortunately, I was busy with some other activities. Nevertheless, I will try to handle that "soon".

szpak · 2020-12-12T22:27:20Z

I wanted to work on it, however, it still fails for me locally with commit 140af32 (your master):

info.solidsoft.gradle.pitest.functional.AcceptanceTestsInSeparateSubprojectFunctionalSpec > should aggregate report from subproject FAILED
    java.lang.IllegalArgumentException: Mutable Project State warnings were found (Set the ignoreMutableProjectStateWarnings system property during the test to ignore):
     - The configuration :for-report:testRuntimeClasspath was resolved without accessing the project in a safe manner.  This may happen when a configuration is resolved from a different project. This behaviour has been deprecated and is scheduled to be removed in Gradle 7.0. See https://docs.gradle.org/6.6/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors for more details.
     - The configuration :shared:testRuntimeClasspath was resolved without accessing the project in a safe manner.  This may happen when a configuration is resolved from a different project. This behaviour has been deprecated and is scheduled to be removed in Gradle 7.0. See https://docs.gradle.org/6.6/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors for more details.
        at nebula.test.IntegrationBase$Trait$Helper.checkForMutableProjectState(IntegrationBase.groovy:114)
        at nebula.test.Integration$Trait$Helper.checkForDeprecations(Integration.groovy:171)
        at nebula.test.Integration$Trait$Helper.runTasks(Integration.groovy:167)
        at nebula.test.Integration$Trait$Helper.runTasksSuccessfully(Integration.groovy:148)
        at info.solidsoft.gradle.pitest.functional.AcceptanceTestsInSeparateSubprojectFunctionalSpec.should aggregate report from subproject(AcceptanceTestsInSeparateSubprojectFunctionalSpec.groovy:31)

21 tests completed, 1 failed, 3 skipped

When I ignored the deprecation warnings it started to fail with:

                Caused by:
                java.lang.IllegalStateException: Failed to build: no lineCoverageFiles have been set
                    at org.pitest.aggregate.ReportAggregator$Builder.validateState(ReportAggregator.java:226)
                    at org.pitest.aggregate.ReportAggregator$Builder.build(ReportAggregator.java:214)
                    at info.solidsoft.gradle.pitest.report.PitestReportAggregator.aggregate(PitestReportAggregator.groovy:38)

For some reasons the CI builds pass. I made a dummy commit in your branch to make sure Travis builds from the same commit, but it stuck (most likely due to that).

Does ./gradlew check funcTest work for you locally? What OS and Java version do you use (I checked with Java 15, 11 and 8 on Linux)?

Update. You seem to migrate already to travis.com and that test passed :-/
(the fail on upload is related to my CD mechanism and you may ignore it - I will apply a workaround before the next version is released)

szpak · 2020-12-12T23:41:08Z

I'm not sure why it was happening only on my machine, but both "no lineCoverageFiles" and deprecation warnings were caused by the fact that in functional tests pitestReportAggregate was executed before the regular pitest tasks. In that situation, there were not files with line coverage which was generating the errors. I added mustRunAfter (mysteriously shouldRunAfter was not enough 🤔 ) which fixed the test.

That brought me to the conslusion that AggregateReportTask has no inputs and outputs, it will be always not up-to-date (even there are not changes in previously generated by pitest tasks reports). Woudn't be possible to try to expose them as fields in AggregateReportTask?

Unfortunately, solving that took a lot of time, so I will experiment with avoiding hardcoded versions of runtime dependencies another time :-/.

…ss for multi-project resolves: szpakgh-80

MikeSafonov · 2020-12-14T14:59:50Z

Greetings. I`ve revorked AggregateReportTask to use Output and Intput

szpak · 2021-01-20T22:44:37Z

My initial approach with compileOnly dependency (and runtime hacks) failed. I asked for other suggestions on their forum.

szpak · 2021-02-28T10:22:56Z

src/main/groovy/info/solidsoft/gradle/pitest/AggregateReportTask.groovy

+    }
+
+    private boolean isRootPitestConfigured() {
+        return project.plugins.hasPlugin(PitestPlugin.PLUGIN_ID) && project.extensions.findByType(PitestPluginExtension)


I'm back to that topic and it looks quite promising :-). However, adjusting the code, I wonder if there is a reason why you for the root project check for both plugin and its extension, but for subproject you only check for the plugin existence?

Extension should be applied automatically when the plugin is applied, so maybe it is superfluous?

(please do not commit any changes, I'm working on dependencies which I plan to push to that branch once ready)

Extension should be applied automatically when the plugin is applied, so maybe it is superfluous?

You are completely right.

With the Worker API: https://docs.gradle.org/nightly/userguide/worker_api.html#changing_the_isolation_mode Also for aggregate reporting.

szpak · 2021-03-01T00:56:27Z

With a help from @marcphilipp, I was able to retain the PIT version configuration at runtime capability using the Worker API.

@MikeSafonov I had to rework your PR a little bit. Please take a look if it looks ok for you (the last commit). I will write some more tests tomorrow and I plan to merge it soon. Feel free to make new commits if needed, but please do not rebase this branch.

MikeSafonov · 2021-03-01T12:16:25Z

With a help from @marcphilipp, I was able to retain the PIT version configuration at runtime capability using the Worker API.

@MikeSafonov I had to rework your PR a little bit. Please take a look if it looks ok for you (the last commit). I will write some more tests tomorrow and I plan to merge it soon. Feel free to make new commits if needed, but please do not rebase this branch.

Congratulations! I have already looked at your changes and they looks good to me.
Looking forward to the new version of the plugin!

szpak · 2021-03-01T23:31:47Z

I've merged it manually into devel and having some other (unrelated) changes implemented I will release 1.6.0 "soon". Thank you for your contribution!

MikeSafonov force-pushed the master branch from 5159cd6 to a270f13 Compare November 29, 2020 12:29

szpak requested changes Nov 29, 2020

View reviewed changes

szpak reviewed Nov 29, 2020

View reviewed changes

...y/info/solidsoft/gradle/pitest/functional/MultiprojectReportAggregationFunctionalSpec.groovy Outdated Show resolved Hide resolved

szpak requested changes Nov 29, 2020

View reviewed changes

refactor: report directory name was extracted into static constant

ddb995f

MikeSafonov force-pushed the master branch from a270f13 to 1db2eb6 Compare November 30, 2020 09:18

szpak requested changes Dec 1, 2020

View reviewed changes

MikeSafonov force-pushed the master branch 2 times, most recently from 0662589 to 140af32 Compare December 2, 2020 07:23

szpak force-pushed the master branch from d1f28d2 to 7ebb588 Compare December 12, 2020 23:31

szpak force-pushed the master branch from 7ebb588 to ba1d13d Compare December 12, 2020 23:43

MikeSafonov force-pushed the master branch 2 times, most recently from e79c141 to d47df23 Compare December 14, 2020 11:17

feat: created new plugin PitestAggregatorPlugin for aggregation proce…

f706426

…ss for multi-project resolves: szpakgh-80

MikeSafonov force-pushed the master branch from d47df23 to f706426 Compare December 14, 2020 13:09

szpak mentioned this pull request Dec 22, 2020

adds testStrengthThreshold config property of PIT 1.6.1 #247

Merged

szpak reviewed Feb 28, 2021

View reviewed changes

[szpak#80] Restore runtime PIT version configuration capability

61f69ec

With the Worker API: https://docs.gradle.org/nightly/userguide/worker_api.html#changing_the_isolation_mode Also for aggregate reporting.

szpak approved these changes Mar 1, 2021

View reviewed changes

[szpak#80] More tests on pitest version in aggregate report

38f606d

szpak force-pushed the master branch from 62be57b to 38f606d Compare March 1, 2021 23:30

szpak added this to the 1.6.0 milestone Mar 3, 2021

szpak merged commit 746313c into szpak:master Mar 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

report aggregation process implemented for multi-project #243

report aggregation process implemented for multi-project #243

MikeSafonov commented Nov 28, 2020

szpak left a comment

szpak Nov 28, 2020

szpak Dec 1, 2020

MikeSafonov commented Dec 11, 2020

szpak commented Dec 12, 2020

szpak commented Dec 12, 2020 •

edited

szpak commented Dec 12, 2020 •

edited

MikeSafonov commented Dec 14, 2020

szpak commented Jan 20, 2021 •

edited

szpak Feb 28, 2021 •

edited

MikeSafonov Mar 1, 2021

szpak commented Mar 1, 2021 •

edited

MikeSafonov commented Mar 1, 2021

szpak commented Mar 1, 2021

report aggregation process implemented for multi-project #243

report aggregation process implemented for multi-project #243

Conversation

MikeSafonov commented Nov 28, 2020

szpak left a comment

Choose a reason for hiding this comment

szpak Nov 28, 2020

Choose a reason for hiding this comment

szpak Dec 1, 2020

Choose a reason for hiding this comment

MikeSafonov commented Dec 11, 2020

szpak commented Dec 12, 2020

szpak commented Dec 12, 2020 • edited

szpak commented Dec 12, 2020 • edited

MikeSafonov commented Dec 14, 2020

szpak commented Jan 20, 2021 • edited

szpak Feb 28, 2021 • edited

Choose a reason for hiding this comment

MikeSafonov Mar 1, 2021

Choose a reason for hiding this comment

szpak commented Mar 1, 2021 • edited

MikeSafonov commented Mar 1, 2021

szpak commented Mar 1, 2021

szpak commented Dec 12, 2020 •

edited

szpak commented Dec 12, 2020 •

edited

szpak commented Jan 20, 2021 •

edited

szpak Feb 28, 2021 •

edited

szpak commented Mar 1, 2021 •

edited