org.gradle.test-report-aggregation restricts on testType

[TWiStErRob]

Gradle 7.6, not sure if this all changed in Gradle 8.0. I can't update yet.

Note: some trivial setup (task group, dependsOn, etc.) is omitted from brevity. The real solution is about ~100 lines.

Context

Here's a use case: I have some production code that works on Java XML/XSLT. There are several implementations for XSLT, and I would like to test them all. The test code is exactly the same for each, so all I need to "parameterize" is the classpath. For this I came up with this combo:

1. Use testing.suites to create integration test tasks, sourceSets and segregate dependencies.
2. Use java.registerFeature to create a shared folder for the common tests.
3. Wire the feature in 2. into each suite's testTask created in 1. .

java.registerFeature("sharedIntegrationTests") { // 2.
	usingSourceSet(sourceSets.create("integrationTest"))
	disablePublication()
}
dependencies { // 2.
	integrationTestImplementation(project)
	integrationTestImplementation(testFixtures(project))
	integrationTestImplementation(libs.test.junit4)
}

testing.suites {  // 1.
	registerIntegrationTest("java") {
		dependencies { /* None, use default from JRE bootclasspath. */ }
	}
	registerIntegrationTest("xalan") {
		dependencies { implementation(libs.xml.xalan) }
	}
	registerIntegrationTest("saxon") {
		dependencies { implementation(libs.xml.saxon) }
	}
}

fun NamedDomainObjectContainerScope<TestSuite>.registerIntegrationTest( // 1.
	name: String,
	configure: JvmTestSuite.() -> Unit
) {
	register<JvmTestSuite>("${name}IntegrationTest") { // 1.
		testType.set("${TestSuiteType.INTEGRATION_TEST}-${name}")
		dependencies {
			implementation(project())
			implementation(testFixtures(project()))
			// essentially `sharedIntegrationTests(project())`, but there's not Kotlin DSL generated for this!
			runtimeOnly(project()) {
				capabilities { // 3.
					requireCapability("${project.group}:${project.name}-shared-integration-tests")
				}
			}
		}
		targets.configureEach {
			testTask.configure {
				systemProperty("net.twisterrob.transform", name)
				testClassesDirs = files(
					testClassesDirs, // Keep original.
					sourceSets["integrationTest"].output.classesDirs, // 3.
				)
			}
		}
		configure()
	}
}

This all works pretty well, my eyes were watering the Gradle DSL design is so beautiful and everything just clicked into place and worked the first time, but then it all turned around when I tried to merge the test reports from the suites. The whole of the org.gradle.jvm-test-suite and org.gradle.test-report-aggregation is built around the assumption that it aggregates the SAME type of tests from different places, but it's not able to merge DIFFERENT test types.

Current Behavior

I want to merge all the suites' results, but they're forcefully separated into testType.set("${TestSuiteType.INTEGRATION_TEST}-${name}").
I literally did a triple-take while reading this sentence, I couldn't believe it:

The type must be unique across all test suites in the same Gradle project.

I cannot set testType.set(TestSuiteType.INTEGRATION_TEST) for all of the suites (which would be the truth), because of the above limitation. And this is where it all breaks when actually trying to use the testType:

reporting.reports.register<AggregateTestReport>("integrationTestAggregateTestReport") {
	testType.set(WhatDoI.PUT_HERE)
}

Expected Behavior (a potential solution)

This is just a guess on how this could be resolved, ideally the whole limitation on uniqueness would be lifted (this is why it's incubating, right?).

reporting.reports.register<AggregateTestReport>("integrationTestAggregateTestReport") {
	// Ideally this wouldn't just be the type, but maybe explicit names of suites?
	testTypes.add("${TestSuiteType.INTEGRATION_TEST}-java")
	testTypes.add("${TestSuiteType.INTEGRATION_TEST}-xalan")
	testTypes.add("${TestSuiteType.INTEGRATION_TEST}-saxon")
}

I spent hours trying to debug and figure out how to do the above, trying hack this line out of TestReportAggregationPlugin:

attributes.attributeProvider(TestSuiteType.TEST_SUITE_TYPE_ATTRIBUTE, report.getTestType().map(tt -> objects.named(TestSuiteType.class, tt)));

I managed to do it (with 1 reflective call), but no luck. I might be missing more steps though.

Workaround

After the hours of trying, I gave up and threw away id(test-report-aggregation) and did it the old way by copying the parts I needed from the plugin:

val integrationTestAggregateTestReport by tasks.registering(TestReport::class) {
	// testResults.from(testing.suites*.targets*.testTask) added later when configuring the test suites.
	destinationDirectory.convention(java.testReportDir.dir("integration-tests/aggregated-results"))
}
testing.suites... targets.configureEach {
	integrationTestAggregateTestReport.configure { testResults.from(testTask) }
}

This feels like I'm hacking, but it works.

Final thoughts

I think this limitation should be fixed in some way for the feature to become stable. The above is just one use case for this. There's another trivial one: I want to create an uber-report from all unit/functional/integration/etc. test suites from all modules. Just one index.html for EVERYTHING. Based on what I've observed here, this is simply not possible today without rewriting the aggregation plugin from scratch.

[ov7a]

Thank you for providing a valid reproducer.

This feature request is in the backlog of the relevant team but this area of Gradle is currently not a focus one so it might take a while before it gets implemented.