StringSpec test cases with multiline names broken

[neelkamath]

Bugs

I think I have found two bugs with StringSpecs. I'm unsure whether other testing styles are affected as well.

Environment

Here's my environment. I'm unsure whether other environments are affected.

• Kotlin/JVM 1.3.72
• Kotest 4.0.5
• Gradle 6.4

Bug 1: Test Failures

Consider this psuedocode.

class TestSuite : StringSpec({
    """
    test
    case
    1
    """ {
        // test
    }

    """
    test
    case
    2
    """ {
        // test
    }

    "test case 3" {
        // test 
    }
})

If I run the entire suite, and the first two test cases fail, then the error message used for one of the cases is used as the error message for the other case as well. Similarly, if only one of the first two cases fail, both fail with the error message of the failed case.

I guess this happens even if there's more than one test case with a multiline name.

Bug 2: Focusing

Focusing (prepending the test case's name with f:) doesn't work for test cases with multiline names. In essence, every test case runs even if I focus a test case with a multiline name.

I guess the same holds true for bangs (prepending the test case's name with !).

[ashishkujoy]

@neelkamath For the first problem if you observe carefully the report mentions 3 test failure instead of 2, with two of them with same stacktrace(and failure message) but only one of them will have actual test name and other will executionError as test name. This is a known problem with reporting in kotest #1330

[ashishkujoy]

Two the second problem
instead of this

"""
f:test
case
2""" {

}

Use

"""f:
test
case
2"""
``` {
}

[sksamuel]

Ashish I think it's worth us calling .trim() on the names when we generate the TestCase instances.

…

On Sat, 16 May 2020 at 11:07, Ashish Kumar Joy ***@***.***> wrote: Two the second problem instead of this """f:testcase2""" { } Use """f:testcase2""" ``` { } — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1441 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFVSGWYB2XBQZGDTJZ5PZTRR222HANCNFSM4NC3HDZA> .

[neelkamath]

@ashishkujoy

Regarding Bug 1

Note: When I refer to "test case", I mean "test case with a multiline name".

I tried to reproduce bug 1 today with simpler code.

When one test case is supposed to pass, and another is supposed to fail, only one fails. This is not a bug.

However, bug 1 still exists, in the other scenario I mentioned. Consider the following code.

import io.kotest.core.spec.style.StringSpec
import io.kotest.matchers.shouldBe

class MyTest : StringSpec({
    """
    test
    case
    1
    """ { 1 shouldBe 2 }

    """
    test
    case 
    2
    """ { true shouldBe false }
})

Both tests are supposed to fail, and they both do. However, the test report shows test case 1's failure for test case 2's failure as well. Please unzip the attached folder and open index.html in your browser to see the Gradle report (I'm unsure whether this is a Gradle bug).

Test Report.zip

Regarding Bug 2

Your focusing technique works, and makes sense. It didn't occur to me that whitespace would've affected it, especially since Gradle replaces multiline test case names with single spaces. It might help to add this to the focus (and bang) docs.

[sksamuel]

This is a gradle limitation whereby it's taking only the first line from the multi line, which is of course just a '\n' in this case, so they clash. We should strip out new lines from multiline strings, as they won't work in intellij or gradle. We already strip newlines in the intellij plugin.

…

On Sat, 16 May 2020 at 20:38, Neel Kamath ***@***.***> wrote: @ashishkujoy <https://github.com/ashishkujoy> Regarding Bug 1 Note: When I refer to "test case", I mean "test case with a multiline name". I tried to reproduce bug 1 today with simpler code. When one test case is supposed to pass, and another is supposed to fail, only one fails. This is not a bug. However, bug 1 still exists, in the other scenario I mentioned. Consider the following code. import io.kotest.core.spec.style.StringSpecimport io.kotest.matchers.shouldBe class MyTest : StringSpec({ """ test case 1 """ { 1 shouldBe 2 } """ test case 2 """ { true shouldBe false } }) Both tests are supposed to fail, and they both do. However, the test report shows test case 1's failure for test case 2's failure as well. Please unzip the attached folder and open index.html in your browser to see the Gradle report (I'm unsure whether this is a Gradle bug). Test Report.zip <https://github.com/kotest/kotest/files/4639514/Test.Report.zip> Regarding Bug 2 Your focusing technique works, and makes sense. It didn't occur to me that whitespace would've affected it, especially since Gradle replaces multiline test case names with single spaces. It might help to add this to the focus (and bang) docs. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1441 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFVSGXALS47EZY5FFIJNATRR4525ANCNFSM4NC3HDZA> .

[sksamuel]

Fix will be included in 4.0.6 and 4.1.0