Error in beforeTest listener is swallowed

[OliverCulleyDeLange]

Kotest 4.0.2
I run a test which throws an exception in my beforeTest{} setup steps, however the process finishes with exit code 0, and no error is printed to console.

Example:

class TestingSpec : StringSpec() {
    init {
        beforeTest {
            throw ExceptionInInitializerError("Fail")
        }

        "test live data updated when doThing()" {
            true shouldBe false
        }
    }
}

Gives:

 "C:\Program Files\Android Studio\jre\bin\java.exe" -ea -Didea.test.cyclic.buffer.size=1048576 "-javaagent:C:\Program Files\Android Studio\lib\idea_rt.jar=55570:C:\Program Files\Android Studio\bin" -Dfile.encoding=UTF-8 -classpath "..." com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit5 uk.co.oliverdelange.wcr_android_kt.TestingSpec
~~~ Kotest Configuration ~~~
-> Parallelism: 1 thread(s)
-> Default test timeout: 600000ms
-> Default test order: TestCaseOrder
-> Default isolation mode: IsolationMode
-> Global soft assertations: False
-> Write spec failure file: False
-> Fail on ignored tests: False
-> Spec execution order: LexicographicSpecExecutionOrder
-> Extensions
  - io.kotest.core.extensions.SystemPropertyTagExtension
  - io.kotest.core.extensions.RuntimeTagExtension
  - io.kotest.core.extensions.IgnoredSpecDiscoveryExtension
  - io.kotest.core.extensions.TagFilteredDiscoveryExtension



Process finished with exit code 0```

Similar to https://github.com/kotest/kotest/issues/307 but thats closed. 

[sksamuel]

There's plenty of tests around this, I will investigate.

[sksamuel]

This is your exact scenario. https://github.com/kotest/kotest/blob/master/kotest-runner/kotest-runner-junit5/src/jvmTest/kotlin/com/sksamuel/kotest/runner/junit5/StringSpecEngineKitTest.kt#L253

[OliverCulleyDeLange]

Interesting, the plot thickens. So i ran that test, in the kotest repo to see if its a platform issue (java version or windows related), and it passes ok.

I then tried to run my failing test through gradle test and it appeared to be fine, the test runs as expected and fails due to exception.

I then ran my failing test again through Android Studio and it worked :/ weird...

I did a gradle clean, and ran the test again through Android Studio and the process finished with exit code 0 with no errors on the console.

Then i tried gradle test again, but it didn't find any changes so didn't run anything. > Task :app:test UP-TO-DATE

Then after doing a clean and test again, my failing test isn't being found anymore, wheras my other tests are. Also, there appears to be a huge discrepancy with what is printed to the console when running via gradle cla, gradle via Android studio and via Android studio run configs.

Gradle version 6.3 here, i don't know enough about gradle to understand this. Which version of gradle do you use so i can try that?

[OliverCulleyDeLange]

Aha, it has to do with the type of exception thrown from the beforeTest block.

RuntimeException works as expected.
ExceptionInInitializerError makes everything go a bit haywire... and this is the exception that was being thrown from my test.

Can you confirm this your side?

I beleive regardless of the type of exception thrown during the setup, the tests should always finish and spit out that exception on the console somehow.

[sksamuel]

I'll need to add more tests. The code catches Throwable so it should catch anything. I don't see why ExceptionInInitializerError would break it.

…

On Tue, 14 Apr 2020 at 02:37, Oliver Culley de Lange < ***@***.***> wrote: Aha, it has to do with the type of exception thrown from the beforeTest block. RuntimeException works as expected. ExceptionInInitializerError makes everything go a bit haywire... and this is the exception that was being thrown from my test. Can you confirm this your side? I beleive regardless of the type of exception thrown during the setup, the tests should always finish and spit out that exception on the console somehow. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1381 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFVSGRIMGQQ5623R47Q6L3RMQHC7ANCNFSM4MHAWJOA> .

[sksamuel]

Related issue from slack.
https://kotlinlang.slack.com/archives/CT0G9SD7Z/p1587556576184500?thread_ts=1587552480.178300&cid=CT0G9SD7Z

[ashishkujoy]

@sksamuel ExceptionInInitializerError extends LinkageError which in jvm platform implementation of Try, we have marked it nonfatal error I think that's the reason it gets swallowed in beforeTest

kotest/kotest-fp/src/commonMain/kotlin/io/kotest/fp/Try.kt

Lines 10 to 14 in e176cc3

	inline operator fun <T> invoke(f: () -> T): Try<T> = try {
	Success(f())
	} catch (e: Throwable) {
	if (nonFatal(e)) Failure(e) else throw e
	}

kotest/kotest-fp/src/jvmMain/kotlin/io/kotest/fp/nonFatal.kt

Lines 3 to 6 in e176cc3

	actual fun nonFatal(t: Throwable): Boolean = when (t) {
	is VirtualMachineError, is ThreadDeath, is InterruptedException, is LinkageError -> false
	else -> true
	}

https://docs.oracle.com/javase/8/docs/api/java/lang/ExceptionInInitializerError.html

[sksamuel]

I'm releasing 4.0.4 with this fix.

…

On Thu, 23 Apr 2020 at 05:01, ashish kumar joy ***@***.***> wrote: @sksamuel <https://github.com/sksamuel> ExceptionInInitializerError extends LinkageError which in jvm platform implementation of Try, we have marked it nonfatal error I think that's the reason it gets swallowed in beforeTest https://github.com/kotest/kotest/blob/e176cc3e14364d74ee593533b50eb9b08df1f5d1/kotest-fp/src/commonMain/kotlin/io/kotest/fp/Try.kt#L10-L14 https://github.com/kotest/kotest/blob/e176cc3e14364d74ee593533b50eb9b08df1f5d1/kotest-fp/src/jvmMain/kotlin/io/kotest/fp/nonFatal.kt#L3-L6 https://docs.oracle.com/javase/8/docs/api/java/lang/ExceptionInInitializerError.html — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1381 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFVSGVNWIQQEG634OTN2WDROAGXDANCNFSM4MHAWJOA> .