Document how to separate test types (unit, integration, ui, end-to-end, etc.) when using Spek

[jcornaz]

Hello,

As far as I understand neither @Category (Junit4) nor @Tag (JUnit5) annotations can be used in conjunction with Spek righ?

Separating test source would be my preferred option. But I encountered trouble trying do to so using gradle-testset-plugin... (see unbroken-dome/gradle-testsets-plugin/issues/62).

The last option I am aware of, is to create filter rules based on the class name. That's my least preferred option, because I find it quite error prone.

May I ask what is the recommended way of separating integration-test and unit-test when using Spek?

Is it planned to support or add an equivalent of @Tag for Spek?

EDIT: Actually even filtering by name doesn't work. Gradle's test filter only work for Junit classes, but has no impact on Spek tests. So, as far as I know, there is no way to separate test type when using Spek...

[jcornaz]

@tkrullmann (from gradle-testset-plugin) answered the following to my problem with Spek:

I tried your setup with Spek and it seems to me that Spek, unlike JUnit, scans the entire classpath for tests, which is probably a bug.

Gradle's Test tasks have two separate properties for this, classpath (which contains the entire classpath) and testClassesDirs (which should be the only place to scan for tests). The Spek runner doesn't seem to respect this.

You can verify this by logging the files in the two properties before the test is executed:

tasks.withType<Test> {
    doFirst {
        println ("""
            Test task "$name":
              testClassesDirs: ${testClassesDirs.joinToString()}
              classpath: ${classpath.joinToString()}
        """.trimIndent())
    }
}

@raniejade, Would it be possible to make the Spek runner to scan only the testClassDirs instead of the entire classpath?

[charleskorn]

Another option is to create separate Gradle source sets and corresponding test tasks. This has the advantage of allowing you to have different dependencies for each type of test (which can be useful when you want to enforce the level at which different tests operate).

I have this set up in a project, take a look at https://github.com/charleskorn/batect/blob/master/app/build.gradle, https://github.com/charleskorn/batect/blob/master/app/gradle/integrationTest.gradle and
https://github.com/charleskorn/batect/blob/master/app/gradle/journeyTest.gradle. Unit tests go in the main test source set, integration tests in integrationTest, journey tests in journeyTest.

[jcornaz]

Another option is to create separate Gradle source sets and corresponding test tasks. This has the advantage of allowing you to have different dependencies for each type of test (which can be useful when you want to enforce the level at which different tests operate).

@charleskorn Yes, that's exactly what I did. And it is also my prefered option.

The only difference is that I use gradle-testsets-plugin to make my build really simpler, and avoid writing the same boiler-plate in all my projects. But Spek doesn't work well with that plugin, because it ignores the testClassDirs and scan the whole classpath

I see, you managed to make it work with a manual configuration. But it is really a shame that we cannot use gradle-testsets-plugin. I'm not supper happy to have maintain all this complex build setup when I could use a plugin.

Anyway, I personally think that:

• Spek should consider testClassDirs instead of scaning the complete classpath.
• There is missing documentation about how to do separate test types when using Spek.

EDIT: gradle-testsets-plugin published a fix, which solve my problem of using Spek with that plugin. I remain, however interested to know what it the "recommended" approach, and I think it deserve to be documented.

[raniejade]

Apologies for the late reply - been very busy lately.

Spek should consider testClassDirs instead of scaning the complete classpath.

It wasn't ignored, nothing was passed to Spek. Scanning the classpath will only happen if no ClasspathRootSelector is passed to it (https://github.com/spekframework/spek/blob/2.x/spek-runner/junit5/src/main/kotlin/org/spekframework/spek2/junit/SpekTestEngine.kt#L44). My guess is gradle's JUnit 5 integration does not pass that selector.

There is missing documentation about how to do separate test types when using Spek

Indeed.

interested to know what it the "recommended" approach, and I think it deserve to be documented.

There's no "recommended" approach, documentation-wise I'd used vanilla gradle (separate sources sets, manually creating test class, etc...).

Looks like the bug was in gradle-testsets-plugin, can I close this issue now?

[jcornaz]

Looks like the bug was in gradle-testsets-plugin, can I close this issue now?

There was indeed a problem in gradle-testsets-plugin. Many thanks to @tkrullmann for solving it.

However, if I'm personally no longer blocked, I don't consider this issue solved.

Here hare the things that (I think) should be done before closing this issue:

• Document that @Tag and filter by class name doesn't work with Spek
  • Or eventually support filter by class name and/or @Tag
• Document that the only way to separate test types is to create separate source sets
  • And eventually give an example (users coming from Junit are used to do that with @Category and @Tag)
• Document how to separate test types for maven users.

[raniejade]

Document that @tag and filter by class name doesn't work with Spek
Or eventually support filter by class name and/or @tag

I think we need to disambiguate in the docs JUnit Platform vs JUnit Jupiter (collectively known as JUnit 5), pretty common misunderstanding.

Document that the only way to separate test types is to create separate source sets
And eventually give an example (users coming from Junit are used to do that with @category and @tag)

Yep, definitely going to happen.

Document how to separate test types for maven users.
I'll leave up to the people who wants to contribute, I don't use and not very familiar with maven.

[pc-fmarin]

I'm trying to use gradle-testsets-plugin to do this, after putting the following simple snippet in my build.gradle the integrationTests are not executed, they are compiled but not executed. Is there something I'm missing in my config?
I'm using gradle 5.1.1
`
plugins {
id 'org.unbroken-dome.test-sets' version '2.1.1'
}

testSets {
integrationTest
}
`

[tkrullmann]

The other test sets are not executed by default - that's by design. You have to either include the integrationTest task when running Gradle, or have something like

check.dependsOn integrationTest

in your build script.

[pc-fmarin]

yeah I got that, I'm running gradle integrationTest and that is not executing tests, after attempting w/ debug I noticed these lines during the testIntergration execution. These tests are kotlin folder

14:46:29.278 [DEBUG] [TestEventLogger] Gradle Test Run :integrationTest STARTED
14:46:29.304 [DEBUG] [org.gradle.api.internal.tasks.testing.detection.ClassFileExtractionManager] extracted class org/jetbrains/spek/api/Spek from spek-api-1.2.1.jar
14:46:29.305 [DEBUG] [org.gradle.api.internal.tasks.testing.detection.AbstractTestFrameworkDetector] test-class-scan : failed to scan parent class java/lang/Object, could not find the class file
14:46:29.306 [DEBUG] [TestEventLogger]
14:46:29.306 [DEBUG] [TestEventLogger] Gradle Test Run :integrationTest PASSED

[pc-fmarin]

Finally got my different sourceSets working. I have to add this to discover and execute the tests.

tasks.withType(Test) {
    // was not finding integration tests without this
    scanForTestClasses = false
    include "**/*Test.class" // whatever Ant pattern matches your test class files
}

Then I had to add the spek framework explicitly to the integrationTest config so the test runner can find the spek tests to execute.

NOTE: I also had to repeat the testing log configuration I had in the test config so the SUCCESS tests appear on console.

integrationTest {
    // necessary for integration cause not inheriting from test
    useJUnitPlatform {
        includeEngines 'spek'
    }
    testLogging.showStandardStreams = true
    testLogging.showExceptions = true
.
.