Running with contracts in different sourceset than "test"

[binkley]

tl;dr -- I'd like to configure the source set used for contract tests

We're introducing SCC into our projects, and would like to place contract tests and the base test class into a different source set than "test". We have our tests segregated into sourcesets by their level in the "test pyramid" (see https://martinfowler.com/articles/practical-test-pyramid.html):

• "test" -- (fastest) unit tests, no Spring injection, only mocking when needed
• "integration" -- Spring tests (think "component integration" within the application), excluding unmocked database
• "database" -- Spring tests with an embedded non-memory database (Postgres), mostly for testing schema-java impedence
• "live" -- (slowest) full app-context tests, solely for checking certain endpoints, such as Swagger or Actuator, are present (but still using an embedded database)

These are defined using the helpful org.unbroken-dome.test-sets plugin.

We'd like to introduce a "contract" source set, which would also be full app-context, and sit atop the test pyramid, just under manual exploratory testing.

It's straight-forward enough to set contractsDslDir to move the contract tests and test base class from src/test to src/contractTest. However, that doesn't change that the SCC tasks are hooked into the "test" source set tasks, eg, https://github.com/spring-cloud/spring-cloud-contract/blob/6464513/spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/groovy/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.groovy#L112 and https://github.com/spring-cloud/spring-cloud-contract/blob/6464513/spring-cloud-contract-tools/spring-cloud-contract-gradle-plugin/src/main/groovy/org/springframework/cloud/contract/verifier/plugin/SpringCloudContractVerifierGradlePlugin.groovy#L89.

One effect of this: we're unable to keep our dependencies "hygienic", for example, adding the embedded database dependency only to tests higher in the pyramid, or avoiding app-context in the unit tests. It also places our slowest tests (contract tests) at the bottom of the pyramid.

Reading through the plugin sources, even with exposing the depends-on for "compileTestJava" as a settable property (so we could change it to "compileContractTestJava"), it's unclear quite how to separate "test" and "contractTest".

---

I tried this:

contracts {
    baseClassForTests = "hm.binkley.basilisk.contracts.ContractTestBase"
    contractsDslDir = file("${project.projectDir}/src/contractTest/resources/contracts")
    generatedTestSourcesDir = file("${project.buildDir}/generated-contractTest-sources/contracts")
    generatedTestSourcesDir = file("${project.buildDir}/generated-contractTest-resources/contracts")
    testFramework = "JUNIT5"
}

which seems ok on the surface, but led to:

> Task :compileTestGroovy
/Users/boxley/src/java/basilisk/build/generated-contractTest-resources/contracts/hm/binkley/basilisk/contracts/BasiliskTest.java:5: error: cannot find symbol
import hm.binkley.basilisk.contracts.ContractTestBase;
                                    ^
  symbol:   class ContractTestBase
  location: package hm.binkley.basilisk.contracts
/Users/boxley/src/java/basilisk/build/generated-contractTest-resources/contracts/hm/binkley/basilisk/contracts/BasiliskTest.java:20: error: cannot find symbol
public class BasiliskTest extends ContractTestBase {
                                  ^
  symbol: class ContractTestBase

Given that I had moved the test base class to src/contractTest, it is clear that the "test" source set classpath was the one used rather than "contractTest".

[marcingrzejszczak]

I was thinking about this issue and came to the following conclusions (there was also an similar issue on SO - https://stackoverflow.com/questions/54547390/generated-test-sources-not-getting-invoked-as-part-of-the-build/54712360#54712360).

We shouldn't be adding the generated test sources folder to the groovy sources, unless you're using Spock. Otherwise it should go to java sources

Your case is extremely custom. I think that what you should do is to add it to your sources manually (more or less in such a way)

sourceSets {
   contractTest.java.srcDirs += new File(project.buildDir, "generated-test-source").toString()
}

that should work I guess. Can you double check that?

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[spring-projects-issues]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[goostleek]

We have the same setup. Adding generated-test-sources to the source set helped with the compilation but fails at runtime. We have

sourceSets {
    integrationTest {
        java.srcDirs file('build/generated-test-sources')
        compileClasspath += main.output + test.output
        runtimeClasspath += main.output + test.output
    }
}
task integrationTest(type: Test) {
    testClassesDirs = sourceSets.integrationTest.output.classesDirs
    classpath = sourceSets.integrationTest.runtimeClasspath
}
configurations {
    integrationTestImplementation.extendsFrom testImplementation
    integrationTestRuntime.extendsFrom testRuntime
}

but executing gradle integrationTests results in

> Task :fib-backend-app:integrationTest

generated.com.lawrencespring.fib_backend.MessagingTest initializationError FAILED

  java.lang.IllegalStateException: Unable to find a @SpringBootConfiguration, you need to use @ContextConfiguration or @SpringBootTest(classes=...) with your test



FAILURE: Executed 1 tests in 7.6s (1 failed)


1 test completed, 1 failed

> Task :fib-backend-app:integrationTest FAILED

FAILURE: Build failed with an exception.

[marcingrzejszczak]

How does your generated test and base class look like?

[ldziedziul]

Here's my example of how to make it work: https://github.com/ldziedziul/contracts-in-custom-sourceset-example