Add Jacoco test coverage support to TestKit Gradle daemons

[mateuszkwiecinski]

My Gradle test runs increased instability from being just flaky to failing at ~100% rate. The exception is:

org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':android:test'.
...
Caused by: java.lang.RuntimeException: Failed to store cache entry for task ':android:test'
...
Caused by: java.lang.RuntimeException: Could not pack tree 'jvmArgumentProviders.jacocoAgent$0.jacoco.destinationFile': java.io.IOException: Request to write '65536' bytes exceeds size in header of '1506326' bytes for entry 'tree-jvmArgumentProviders.jacocoAgent%240.jacoco.destinationFile'
...
Suppressed: java.io.IOException: This archive contains unclosed entries.
...
Caused by: java.io.UncheckedIOException: java.io.IOException: Request to write '65536' bytes exceeds size in header of '1506326' bytes for entry 'tree-jvmArgumentProviders.jacocoAgent%240.jacoco.destinationFile'
...
Caused by: java.io.IOException: Request to write '65536' bytes exceeds size in header of '1506326' bytes for entry 'tree-jvmArgumentProviders.jacocoAgent%240.jacoco.destinationFile'

The build passes if caching is disabled
edit: Actually, after disabling the cache build becomes flaky again and sometimes fails on jacocoTestReport task with

Caused by: : Unable to read execution data file /home/runner/work/project-starter/project-starter/android/build/jacoco/test.exec
at org.jacoco.ant.ReportTask.loadExecutionData(ReportTask.java:518)
at org.jacoco.ant.ReportTask.execute(ReportTask.java:491)
...
Caused by: java.io.EOFException
at org.jacoco.core.internal.data.CompactDataInput.readBooleanArray(CompactDataInput.java:64)
at org.jacoco.core.data.ExecutionDataReader.readExecutionData(ExecutionDataReader.java:150)
...

Expected Behavior

I expect build to complete successfuly

Current Behavior

Build fails with exception pasted above.

Context

I started observing this kind of exceptions in gradle 6.7, and I assumed that's something on my side but didn't have time to investigate. Now, when I'm using most recent Gradle version 6.8.3, I found a bit of time and discovered that disabling Gradle caching the build passes (both locally and on a GA runner)

Steps to Reproduce

set org.gradle.caching=true in gradle.properties, run any test task.

Your Environment

Build scan URL: https://scans.gradle.com/s/nfobgvarfutg2
Failer run with full stacktrace: https://github.com/usefulness/project-starter/pull/159/checks?check_run_id=2160476298

[ljacomet]

@gradle/execution This looks like a large entry handling issue

[wolfs]

Do you have multiple tasks writing to the same Jacoco destination file? The thing you are seeing may be caused by another task writing to the destination file while Gradle tries to package the file.

[mateuszkwiecinski]

To be honest I'm don't know how Jacoco integration with test tasks works. You're right and there are multiple test tasks (in a project with Jacoco plugin applied) running in paralell when I experience the failure.

You can find my jacoco config here: https://github.com/usefulness/project-starter/blob/restore_gradle_caching/jvm/src/main/kotlin/com/project/starter/modules/internal/KotlinCoverage.kt#L11 if it causes Jacoco to write to a single destination file then I'm not aware of that fact.
According to the documentation I'm not doing much more than "standard plugin configuration".

I printed out all destination files using

pluginManager.withPlugin("jacoco") {
        tasks.withType(Test) {
            logger.quiet("destinationFile for project=$project.path task=$it.path is \${projectDir}/${project.rootDir.relativePath(jacoco.destinationFile)}")
        }
    }

which produces unique destination files:

> Configure project :android
destinationFile for project=:android task=:android:test is ${projectDir}/android/build/jacoco/test.exec

> Configure project :config
destinationFile for project=:config task=:config:test is ${projectDir}/config/build/jacoco/test.exec

> Configure project :jvm
destinationFile for project=:jvm task=:jvm:test is ${projectDir}/jvm/build/jacoco/test.exec

> Configure project :quality
destinationFile for project=:quality task=:quality:test is ${projectDir}/quality/build/jacoco/test.exec

> Configure project :versioning
destinationFile for project=:versioning task=:versioning:test is ${projectDir}/versioning/build/jacoco/test.exec

but I'm not sure if that's what you were asking for.

I can share another build scan https://scans.gradle.com/s/jlunjybfgz2fk I just created locally and github action's failed run with full stacktrace: https://github.com/usefulness/project-starter/pull/206/checks?check_run_id=2403119162

[wolfs]

Do you run the Gradle builds in your test tasks also with Jacoco enabled? I suppose those JVMs will still be around and keep writing to the exec files even after the tests finished.

[mateuszkwiecinski]

Do you run the Gradle builds in your test tasks also with Jacoco enabled?

I think I do 🤔
I'm using https://github.com/koral--/jacoco-gradle-testkit-plugin plugin which, accoring to its documentation, gets applied to test tasks builds here: https://github.com/usefulness/project-starter/blob/restore_gradle_caching/testing/src/main/kotlin/com/project/starter/WithGradleProjectTest.kt#L29

[wolfs]

I suppose this means that you need to stop the daemons started by testkit after running your tests before Gradle tries to package the build cache entry.

[mateuszkwiecinski]

Huh, can you point me at some resources which would allow me to better understand your suggestion? 😅 I'm not sure what kind of deamons testkit starts. How can I interact with them? Whose lifecycle I should I hook up to t start killing them? At which point I do unusual things? Why these daemons aren't stopped automatically?
Should I start calling GradleRunner with --stop argument? Or add a --no-daemon argument?

[wolfs]

I'm not sure what kind of deamons testkit starts.

Testkit (aka GradleRunner) starts regular Gradle daemons.

How can I interact with them? Whose lifecycle I should I hook up to t start killing them? At which point I do unusual things? Why these daemons aren't stopped automatically?

Hmm..., it seems like testkit actually stops the daemons automatically: https://docs.gradle.org/current/userguide/test_kit.html#sec:controlling_the_build_environment

I wonder if that works and I have the wrong idea what is causing the problem or the stopping of the daemons does not happen/work...

[gmazzo]

Do you have multiple tasks writing to the same Jacoco destination file? The thing you are seeing may be caused by another task writing to the destination file while Gradle tries to package the file.

I'm pretty sure the problem is in here:
https://github.com/koral--/jacoco-gradle-testkit-plugin/blob/master/src/main/kotlin/pl/droidsonroids/gradle/jacoco/testkit/JacocoTestKitExtension.kt#L22

It's taking the destinationFile from the Test task it's going to be overridden (or at least modified) because it runs twice (at least): once by the Test task and another by each GradleRunner's run you have inside a test.

[dgeissl]

@gmazzo as I see it the problem is not about 2 jacoco agents writing to one file (as they are configured to append data and this is working as without problems). So writing 2 different files won't fix the problem at hand.

What I've observed is that the test task think's it is finished and then the output caching starts to zip the exec file, but only after that started the testing Gradle VM is terminated and appends data to the exec file. This changes the files size and thus breaks the zipping.

I am not sure if killing the deferred testing vm won't result in jacoco processes colliding in the one output file, but currently there's another problem at hand.

We've been pointed to

gradle/build-logic/cleanup/src/main/kotlin/gradlebuild/cleanup/services/DaemonTracker.kt

Line 35 in 4817230

abstract class DaemonTracker : BuildService<DaemonTracker.Params>, AutoCloseable {

but didn't have time to investigate how to use it and if that consistently fixes the problem.

For now 90% of the time just waiting one or two seconds in doLast also works.

[gmazzo]

@dgeissl you are totally right. Actually yesterday I've opened this https://github.com/koral--/jacoco-gradle-testkit-plugin/pull/23 to revert the change.

To simplify, the problem is that the GradleRunner lives longer than the Test task itself, and when you enable jacoco for it, it modifies the output .exec file while the Build Cache process is generating the cache bundle.

[wolfs]

We don't support running Jacoco coverage on the Gradle daemon, so I'll close the issue right now.

[dgeissl]

@wolfs what is the reasoning behind this descision?
What closing this thicket with this comment just says is "the gradle plugin testframework will not support code coverage".

Did I miss the documentation to tell plugin authors how to do it properly with the right tooling?
I mean most of the code we write nowadays isn't even unit testable -just think of all the abstract getters, injections, not to forget about all the cache and update checks or incremental build features.

Gradle can't seriously promote developing plugins without any test coverage support?

[wolfs]

@dgeissl Sorry about sounding harsh in my comment above.

Currently, running code coverage for Gradle Daemons launched by test kit does not work. You saw that in the ticket above. So that is a missing feature of test kit. I don't think we have somewhere documented that this doesn't work.

Gradle strongly encourages plugin authors to write tests for their plugins. We did not think much about code coverage support for plugin development, though.

How about you open a feature request that Gradle TestKit should support Jacoco code coverage for the code running in the Gradle Daemon? So we have this captured and folks can vote on the issue?

[dgeissl]

@wolfs thanks for your reply. The question for me is why does one have to make a new ticket?

This ticket documents exactly what the problem is. Many affected users are registered and can follow the discussion and the gradle devs will get quick feedback when required. It not like github has completely different issue type with different workflows.

Replacing a label (if you like dispatch this as bug but as feature request) sends a completely different message to your users than just closing the issue.

[wolfs]

Fair enough. I changed the title and reopened the issue.

[EarthCitizen]

I got this issue with Gradle 8.1.1, running test kit, testing a plugin, and also running the above jacoco-gradle-testkit-plugin version 1.0.12. It seems to be random, as when I ran the pipeline stage again, it did not happen a second time.

[pantherdd]

We have a working solution to using JaCoCo with TestKit here: https://discuss.gradle.org/t/jacoco-gradle-test-kit-with-java/36603/10 (maybe the JaCoCo plugin could do the same?)