Test Case Config does not behave as I expected

[Jager567]

The documentation states that I can add a timeout to a test. It states that it is "Useful for code that is non-deterministic and might not finish". This leads me to believe that this setting will stop the test execution after the given duration has passed. However, when I accidentally executed a test that called a method with an infinite loop, it didn't stop the execution after the timeout had passed. Is this a documentation issue, a fault on my side or a bug?

After some testing, I found out that this fails, but only after the full test has finished:

class FiniteLoopString: StringSpec() {
    "has a long finite loop test".config(timeout = 100.milliseconds) {
        val startTime = currentTimeMillis()
        while (currentTimeMillis() < startTime + 1000) {
            "this" shouldNotBe "that"
        }
    }
}

while the following test just keeps running forever:

class InfiniteLoopString: StringSpec() {
    "has an infinite loop test".config(timeout = 100.milliseconds) {
        while (true) {
            "this" shouldNotBe "that"
        }
    }
}

During my testing, I also found that this test succeeds, even though it takes longer than the timeout:

class SuccessfulTest: WordSpec() {
    init {
        "String.length" should {
            "return the length of the string".config(timeout = 2.seconds) {
                val startTime = currentTimeMillis()
                while (currentTimeMillis() < startTime + 10000) {
                    "this" shouldNotBe "that"
                }
            }
        }
    }
}

[sksamuel]

Definitely a bug. The timeout should kick in for any test that takes longer than the timeout.

[sksamuel]

Which version of kotlintest ?

[Jager567]

I tested this with version 3.1.10.

[LeoColman]

I'll grab this if you're not working on it, @sksamuel .

[sksamuel]

I'll fix this as part of #518 as it will need to work differently.

[sksamuel]

Added. Review as part of #518

[sksamuel]

Please try 3.2.0-RC1 as this should be fixed now. @Jager567

[Jager567]

It seems that test cases now fail correctly, but they still aren't aborted when they haven't finished executing within the timeout period. This means that

class SuccessfulTest: WordSpec() {
	init {
		"String.length" should {
			"return the length of the string".config(timeout = 2.seconds) {
				val startTime = currentTimeMillis()
				while (currentTimeMillis() < startTime + 10000) {
					"this" shouldNotBe "that"
				}
			}
		}
	}
}

now correctly fails, but only after having executed for 10 seconds, while a test with an infinite loop will never finish.

[sksamuel]

A normal test should abort if it's interruptable.

The problem is how do you interrupt a while loop.
You can't.

I can inject a thread interrupt.
I can cancel a coroutine.

But if you want to use an unbounded while loop then you're on your own.

[Jager567]

Because the documentation states that adding a timeout is useful for code that might not finish, I expected that adding a timeout to my test would create something similar to JUnit5 assertTimeoutPreemptively.

That JUnit function does exactly what you claim to be impossible: when it receives an infinite while loop, it will stop the execution when the specified time has passed. It does so by running the test in a new thread. When the timeout has passed, it stops the thread regardless of its state.

I understand that the timeout kotlintest provides is not implemented in the same way. What I do want to suggest is that something is added to the documentation stating that timeouts can only interrupt things that are interruptable.

[sksamuel]

Ok, if junit is doing it, then it's probably using Thread.stop, which I can do as well, but it's been deprecated since Java 1.1. That will actually "kill" a thread in the way I said was impossible. I should have said "impossible without hacky solutions" :)

I suppose we could do the same in theory. Let me look into their implementation.

[sksamuel]

And you're right, at the very least the docs should be clear.

[sksamuel]

So JUnit's implementation is surprising to me.

They just wait xx seconds, and if you're not finished, they'll interrupt your thread, and then carry on, reporting it as failed. So your test report is all hunky dory.

Your thread won't actually be killed though, and it'll just continue in the background until the JVM shuts down. I suppose they are either naive, or assume that the JVM will terminate after the tests complete.

I'm not sure which is better.

@Kerooker thoughts?

[sksamuel]

In 3.2.1 the behavior is now the same as JUnit.