Demonstrate how to merge Jacoco reports from different sub-projects

[lacasseio]

Expected Behavior

Users are left figuring out how this is done from the documentation or by searching the web.

Current Behavior

It is not obvious to users how this can be achieved. It seems like a common scenario that we should demonstrate.

[vlsi]

In practice, merging is complicated.

Consider the build consists of core and ui modules (ui uses core).
Consider core module lacks tests (e.g. no tests at all), and ui has some tests.

Of course ui uses core, so when ui tests are executed some core classes are "covered". Should it be represented as "covered"?

So it looks like there are at least two ways to approach that:

1. Merge all the coverage information from all the executions. It helps to identify "completely uncovered" lines, however it would produce false sense of having good coverage when low-level APIs are covered in E2E tests only.

2. Merge the coverage, however make sure that ui tests add coverage info only for ui package. Then the report could identify that core module lacks tests.

[vlsi]

A connected piece is sending the coverage information to https://sonarcloud.io
https://github.com/apache/jmeter/blob/7148f87e686f5c1276c70cbfe15445e0feef79f7/build.gradle.kts#L197-L200

A connected piece is merging coverage from custom-made javaexec tasks (== integration tests that launch processes).
A bit of a problem with custom tasks is current API lacks extensions.withType<JacocoTaskExtension>.configureEach {...}.
JaCoCo extension is added after creating the task, so one needs to use afterEvaluate or something like that.
https://github.com/apache/jmeter/blob/7148f87e686f5c1276c70cbfe15445e0feef79f7/build.gradle.kts#L279-L281

[lacasseio]

Thanks @vlsi for adding more information on this. We are hearing the complaint about the documentation and we hope to start moving forward with focused changes that address common pain in build scripts. This input you are providing is very valuable for us to write the best samples for users but also address any issue those common use cases cause to our users.

[vlsi]

One more data point: JaCoCo generates .exec files, and it is common misconception that is "the raw data". As @Godin says, .exec are too low-level, non-portable binaries, and they must not be used by the end users (they should avoid tie their workflows to .exec files too much). For instance, they should always produce either xml (for tools like sonarcloud) or html (for UI).

The catch for me there was I thought of "reports" for human processing. I thought "JaCoCo xml report" was meant for the case when I would like to process it myself with my own tool.

On contrary, XML "reports" are very often mandatory for external tools (e.g. coverage, bugs, etc), so it might make sense to drop a note like if you are going to send coverage data to external systems for visualization, it is likely you need xml report configured for that to JaCoCo Report configuration section.

[wolfs]

See also the discussion here and mockito/mockito#1699.

[pioterj]

Documenting how to do this / creating a sample is the first step. At the end of the day, it should be easier as it's such a common use case.

[fieldju]

What is the state of this, it seems as if it is broken as of 6.0

I can seem to do what is outline here with 6.0.1: https://gist.github.com/aalmiray/e6f54aa4b3803be0bcac#file-build-gradle-L31-L42

I get

> Cannot set the value of read-only property 'executionData' for task ':generateMergedReport' of type org.gradle.testing.jacoco.tasks.JacocoReport.

ok digging through 5x vs 6x figured this out and it seems to work

task generateMergedReport(type: JacocoReport) {
  dependsOn = subprojects.test
  additionalSourceDirs.setFrom files(subprojects.sourceSets.main.allSource.srcDirs)
  sourceDirectories.setFrom files(subprojects.sourceSets.main.allSource.srcDirs)
  classDirectories.setFrom files(subprojects.sourceSets.main.output)
  executionData.setFrom project.fileTree(dir: '.', include: '**/build/jacoco/test.exec')
  reports {
    xml.enabled false
    csv.enabled false
    html.enabled true
  }
}

[bmuskalla]

Starting with 6.4, we added a sample showing how to properly configure a multi-module build using jacoco (HTML or XML for Sonar). As mentioned above already, xml is the preferred (and since Sonar Scanner 6.0 only) way to send code coverage information to Sonar. Another gotcha to look out for is that jacoco reports need to be generated before any of the sonar tasks run (see also SONARGRADL-68 and SONARGRADL-59