Crashed tests are not reported as failures

[rjra100]

If a test process crashes altogether (for example, segfaults) the explorer appears to ignore it rather than marking it as failed. The test retains the "not run" icon, and if viewing the tests as a tree view, the icon on the containing suite ignores the result (so if all the other tests succeed and the tree is collapsed, there may be no visual indication that any test has failed).

In case it matters, our tests are mostly C++ using GTest and registered via cmake's gtest_discover_tests function, but I think it'd happen regardless of the test framework.
With a test suite like

TEST(TestSuite, TestCrashingFunction)
{
    int x = *(reinterpret_cast<int*>(0));
    (void)x;
    GTEST_SUCCEED();
}
TEST(TestSuite, TestAllIsWell)
{
    GTEST_SUCCEED();
}

CTest itself reports something like

Test project /build/Debug
    Start 125: TestSuite.TestCrashingFunction
1/2 Test #125: TestSuite.TestCrashingFunction ...***Exception: SegFault  0.00 sec
    Start 126: TestSuite.TestAllIsWell
2/2 Test #126: TestSuite.TestAllIsWell ..........   Passed    0.00 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.02 sec

The following tests FAILED:
        125 - TestSuite.TestCrashingFunction (SEGFAULT)
Errors while running CTest

But the test explorer shows

EDIT:
Sorry, forgot to mention the environment. I reproduced this in VSCode 1.63.2 on Windows, CMake Test Explorer 0.15.1 (with Test Explorer UI 2.21.1) and developing in either a WSL2+Docker environment via Remote-Containers 0.209.6 or a remote server via Remote-SSH 0.70.0). I'm told the behaviour is the same on a Mac + Docker Desktop setup. The actual dev environment is Linux (RHEL7) in all cases. I've seen the same behaviour for some time, though, so I don't think this is new or version-specific.

[fredericbonnet]

Thanks for reporting! I've found and fixed the error, it will be part of the next patch release.

The error lies in the file src\cmake-runner.ts at line 32:

/** Regexp for test failed line */
const CTEST_FAILED_RE =
  /^\s*\d+\/\d+ Test\s+#(\d+): (\S+).*\.\.\.+\*\*\*Failed/;

The regexp only matches regular failures and not exceptions, the fix is to simply remove the trailing Failed so as to match all causes of failures.

[rjra100]

That's great; thanks!

[fredericbonnet]

Latest version 0.15.2 should fix the issue, can you confirm it works on your side?

[rjra100]

Looks good to me. Thanks very much for the quick turnaround (and sorry it took me a few days to check!)

[fredericbonnet]

No worries!