Failing test running on Kotlin/Native fails with unhelpful error message when run from Gradle command line

[charleskorn]

Which version of Kotest are you using?
5.0.0.419-SNAPSHOT

I have a very simple failing test:

import io.kotest.core.spec.style.ShouldSpec
import io.kotest.matchers.shouldBe

class HelloWorldTest : ShouldSpec({
    should("be able to do arithmetic") {
        (1+2) shouldBe 2
    }
})

If I run this test on the JVM with Gradle (./gradlew jvmTest), I get the following, very clear, error message:

HelloWorldTest[jvm] > should be able to do arithmetic[jvm] FAILED
    io.kotest.assertions.AssertionFailedError: expected:<2> but was:<3>
        at app//HelloWorldTest$1$1.invokeSuspend(HelloWorldTest.kt:8)
        at app//HelloWorldTest$1$1.invoke(HelloWorldTest.kt)
        at app//HelloWorldTest$1$1.invoke(HelloWorldTest.kt)

However, if I run the same test with Kotlin/Native on macOS with Gradle (./gradlew macosX64Test), I get this error message:

HelloWorldTest.should be able to do arithmetic FAILED
    io.kotest.assertions.AssertionFailedError at /opt/buildAgent/work/c5a36d4d82b914cf/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Throwable.kt:24

(I haven't tried it on other platforms.)

Weirdly, if I run the same Gradle task from within IntelliJ (eg. by going to the Gradle window, right-clicking the macosX64Test task and clicking Run), I get a clear error message:

io.kotest.assertions.AssertionFailedError: expected:<2> but was:<3>
Expected :2
Actual   :3
<Click to see difference>

io.kotest.assertions.AssertionFailedError: expected:<2> but was:<3>
	at kotlin.Throwable#<init>(/opt/buildAgent/work/c5a36d4d82b914cf/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Throwable.kt:24)
	at kotlin.Error#<init>(/opt/buildAgent/work/c5a36d4d82b914cf/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Exceptions.kt:12)
	at kotlin.AssertionError#<init>(/opt/buildAgent/work/c5a36d4d82b914cf/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/Exceptions.kt:128)
	at io.kotest.assertions.AssertionFailedError#<init>(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/desktopMain/kotlin/io/kotest/assertions/Exceptions.kt:24)
	at io.kotest.assertions.Exceptions#createAssertionError(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/desktopMain/kotlin/io/kotest/assertions/Exceptions.kt:17)
	at io.kotest.assertions#failure(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/commonMain/kotlin/io/kotest/assertions/failures.kt:42)
	at io.kotest.assertions#failure$default(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/commonMain/kotlin/io/kotest/assertions/failures.kt:40)
	at io.kotest.assertions.eq.NumberEq#equals(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/commonMain/kotlin/io/kotest/assertions/eq/NumberEq.kt:11)
	at io.kotest.assertions.eq#eq(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/commonMain/kotlin/io/kotest/assertions/eq/Eq.kt:30)
	at io.kotest.assertions.eq#eq$default(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/commonMain/kotlin/io/kotest/assertions/eq/Eq.kt:19)
	at io.kotest.matchers#shouldBe__at__0:0(/Users/runner/work/kotest/kotest/kotest-assertions/kotest-assertions-shared/src/commonMain/kotlin/io/kotest/matchers/should.kt:16)
	at HelloWorldTest.$<init>$lambda-1$lambda-0COROUTINE$0.invokeSuspend#internal(/<redacted>/lib/src/commonTest/kotlin/<redacted>/HelloWorldTest.kt:8)
	at HelloWorldTest.$<init>$lambda-1$lambda-0COROUTINE$0.invoke#internal(/<redacted>/lib/src/commonTest/kotlin/<redacted>/HelloWorldTest.kt:7)
	at io.kotest.engine.test.TestCaseExecutor.$execute$lambda-0COROUTINE$82.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/TestCaseExecutor.kt:52)
	at io.kotest.engine.test.TestCaseExecutor.$execute$lambda-0COROUTINE$82.invoke#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/TestCaseExecutor.kt:51)
	at io.kotest.engine.test.extensions.CoroutineScopeTestExecutionFilter.$execute$lambda-1$lambda-0COROUTINE$87.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/CoroutineScopeTestExecutionFilter.kt:18)
	at io.kotest.engine.test.extensions.CoroutineScopeTestExecutionFilter.$execute$lambda-1$lambda-0COROUTINE$87.invoke#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/CoroutineScopeTestExecutionFilter.kt:17)
	at kotlinx.coroutines.intrinsics#startUndispatchedOrReturn__at__kotlinx.coroutines.internal.ScopeCoroutine<0:0>(/opt/buildAgent/work/c5a36d4d82b914cf/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/coroutines/intrinsics/IntrinsicsNative.kt:71)
	at kotlinx.coroutines#coroutineScope(/opt/buildAgent/work/c5a36d4d82b914cf/kotlin/libraries/stdlib/src/kotlin/contracts/ContractBuilder.kt:153)
	at io.kotest.engine.test.extensions.CoroutineScopeTestExecutionFilter.$execute$lambda-1COROUTINE$88.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/CoroutineScopeTestExecutionFilter.kt:17)
	at io.kotest.engine.test.extensions.CoroutineScopeTestExecutionFilter.$execute$lambda-1COROUTINE$88.invoke#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/CoroutineScopeTestExecutionFilter.kt:16)
	at io.kotest.engine.test.TestCaseExecutor.$execute$<anonymous>_4COROUTINE$83.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/TestCaseExecutor.kt:57)
	at io.kotest.engine.test.TestCaseExecutor.$execute$<anonymous>_4COROUTINE$83.invoke#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/TestCaseExecutor.kt:57)
	at io.kotest.engine.test.extensions.GlobalSoftAssertTestExecutionFilter.$execute$lambda-0COROUTINE$91.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/GlobalSoftAssertTestExecutionFilter.kt:23)
	at io.kotest.engine.test.extensions.GlobalSoftAssertTestExecutionFilter.$execute$lambda-0COROUTINE$91.invoke#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/GlobalSoftAssertTestExecutionFilter.kt:22)
	at io.kotest.engine.test.TestCaseExecutor.$execute$<anonymous>_4COROUTINE$83.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/TestCaseExecutor.kt:57)
	at io.kotest.engine.test.TestCaseExecutor.$execute$<anonymous>_4COROUTINE$83.invoke#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/TestCaseExecutor.kt:57)
	at io.kotest.engine.test.extensions.AssertionModeTestExecutionFilter.$execute$lambda-0COROUTINE$85.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/AssertionModeTestExecutionFilter.kt:31)
	at io.kotest.engine.test.extensions.AssertionModeTestExecutionFilter.$execute$lambda-0COROUTINE$85.invoke#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/extensions/AssertionModeTestExecutionFilter.kt:30)
	at io.kotest.engine.test.TestCaseExecutor.$execute$<anonymous>_4COROUTINE$83.invokeSuspend#internal(/Users/runner/work/kotest/kotest/kotest-framework/kotest-framework-engine/src/commonMain/kotlin/io/kotest/engine/test/TestCaseExecutor.kt:57)

When I invoke the tests from the command line, the HTML test report that Gradle generates includes the full error message and stack trace, just like IntelliJ does - so this appears to just be an issue with how the exception is displayed by Gradle on the command line.

This is what my Gradle configuration looks like:

import org.gradle.api.tasks.testing.logging.TestExceptionFormat
import org.gradle.api.tasks.testing.logging.TestLogEvent

plugins {
    kotlin("multiplatform") version "1.5.30"
    id("io.kotest.multiplatform") version "5.0.0.5"
}

repositories {
    mavenCentral()
    maven("https://oss.sonatype.org/content/repositories/snapshots")
}

val kotestVersion = "5.0.0.419-SNAPSHOT"

kotlin {
    jvm()
    linuxX64()
    macosX64()
    mingwX64()

    sourceSets {
        val commonTest by getting {
            dependencies {
                implementation("io.kotest:kotest-assertions-core:$kotestVersion")
                implementation("io.kotest:kotest-framework-api:$kotestVersion")
                implementation("io.kotest:kotest-framework-engine:$kotestVersion")
            }
        }

        val jvmTest by getting {
            dependencies {
                implementation("io.kotest:kotest-runner-junit5:$kotestVersion")
            }
        }

        all {
            languageSettings {
                progressiveMode = true
            }
        }
    }
}

tasks.named<Test>("jvmTest") {
    useJUnitPlatform()

    testLogging {
        showExceptions = true
        showCauses = true
        showStandardStreams = true
        events = setOf(TestLogEvent.FAILED, TestLogEvent.SKIPPED)
        exceptionFormat = TestExceptionFormat.FULL
    }
}

Is this something I need to configure, or is this an issue with Gradle?

[sksamuel]

The MPP, and especially native, support is beta as we build it out for the 5.0.0 release. This kind of feedback is great to help improve the support.

[sksamuel]

I believe this is working for the latest snapshot.

[charleskorn]

Yes it is - thanks for fixing this issue!