"Test results not found" for successful CTests from a CMakeLists file

[FunkyGibbon1711]

Brief Issue Summary

I have a set of tests (ran with CTest) running from a CMakeLists.txt file. The tests run successfully and I can see them reporting as "Passed" in the output window at the bottom of the VSCode window. However, the extension reports them as failing with the message: "Test results not found"

I noticed my VSCode extension had updated overnight to v1.18.41 from v1.17.17 and the issue does not occur when reinstalling the previous latest version. The issue also persists with the pre-release v1.19.0 version

My ctest command is running with the -T test flag as described here: #3150

CMake Tools Diagnostics

No response

Debug Log

No response

Additional Information

No response

[gcampbell-msft]

@FunkyGibbon1711 Could you provide a minimal repro of the problem so that we can investigate? It'd be good to understand your project as well as what your current settings are related to testing. Thanks!

[gcampbell-msft]

@FunkyGibbon1711 One other thing that would be useful if you can't provide a minimal repro, would be to try to install older pre-release versions to try and find what specific pre-release version introduced the change so that we could investigate.

What are your settings for the following: cmake.ctestArgs, cmake.ctestDefaultArgs, cmake.ctest.testExplorerIntegrationEnabled, and cmake.ctest.allowParallelJobs?

[gcampbell-msft]

@FunkyGibbon1711 The main PR that modified behavior for handling testing in the CMake Tools was #3577, I believe. WIth that in mind, if you could compare pre-release versions after 1.18.27, to versions before 1.18.27, that could be a place to start.

Just a reminder, if you could provide us with a minimal repro, then we can handle all of this version investigation. Thanks!

[v-frankwang]

@FunkyGibbon1711 I can't reproduce your problem using a simple project, I'm not sure if I'm reproducing the steps incorrectly, here are my reproduction steps can you give me some of your suggestions?

ENV:
VS Code: 1.88.1
CMake tools: 1.18.29
C/C++:1.20.2

Repro steps recording:

1. Create a folder named "CMakeProject1" and open it with vscode.
2. Add to it a CMakeLists.txt file containing the following code:

cmake_minimum_required(VERSION 3.8) 
project(CMakeProject1)
add_executable(CMakeProject1 "CMakeProject1.cpp" "CMakeProject1.h")
enable_testing()
add_executable(UnitTest "test.cpp")
add_test(UnitTest0 UnitTest 0)
add_test(UnitTest1 UnitTest 1)

3. Add to it a CMakeProject1.cpp file containing the following code:

#include "CMakeProject1.h"
using namespace std;
int main()
{
       cout << "Hello CMake." << endl;
       return 0;
}

4. Add to it a test.cpp file containing the following code:

#include <iostream> 
auto main() -> int { std::cout << "hello world" << std::endl; }

5. Add to it a CMakeProject1.h file containing the following code:

#pragma once
#include <iostream>

6. Run the CMake: configure command.
7. Click "CMakeProject1.cpp" file run the"C/C++:Add Debug Configure" command.
8. Select "C/C++:cl.exe build and debug active file"
9. Click on the testing button in the right navigation bar and then click on Run tests in the Test Explorer window.

Actual results：

[FunkyGibbon1711]

@FunkyGibbon1711 One other thing that would be useful if you can't provide a minimal repro, would be to try to install older pre-release versions to try and find what specific pre-release version introduced the change so that we could investigate.

What are your settings for the following: cmake.ctestArgs, cmake.ctestDefaultArgs, cmake.ctest.testExplorerIntegrationEnabled, and cmake.ctest.allowParallelJobs?

I have tried CMake tools 1.18.26 as you suggested and the problem does not persist. It is only for versions later than 1.18.27 that it occurs. These are values for the variables you have requested:

cmake.ctestArgs: ""
cmake.ctestDefaultArgs: "-T test --output-on-failure"
cmake.ctest.testExplorerIntegrationEnabled: true
cmake.ctest.allowParallelJobs: false

I am using VS Code 1.85.1, GCC 10.5.0 and cmake version 3.29.3

The project is a C project I am developing in my workplace. While I can't give specifics, these are the c_cpp_properties in the project:

    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/**"
            ],
            "defines": [],
            "cStandard": "c11",
            "cppStandard": "gnu++14",
            "intelliSenseMode": "linux-gcc-x64",
            "configurationProvider": "ms-vscode.cmake-tools"
        }
    ],
    "version": 4
}```

and uses the standard CMake tasks.json file. There is an additional setting of `cmake.configureOnOpen: false` used as well (though I doubt this would be impacting this issue)

[gcampbell-msft]

@FunkyGibbon1711 Unfortunately, @v-frankwang and I were not able to reproduce the issue with a simple test project. Is there any way you could create a minimal repro that exemplifies the problem?

[FunkyGibbon1711]

Hi, apologies for the delay in this. I was trying to make a sanitised version of my workspace to give as simple an overview as possible. The issue here occurs when attaching fixtures to tests with the previously mentioned versions of CMakeTools, I have attached an archive of my example project here. This project currently only compiles on x86, some changes will be needed to get linking working if using an ARM Mac:

testProject.zip

[gcampbell-msft]

@FunkyGibbon1711 No worries!

To confirm, you reproduce this on a Mac? If you could provide a screenshot or a gif of the issue as well, that could be helpful. I understand it may not be possible, but if it is, that would be great.

Thanks,

[gcampbell-msft]

@FunkyGibbon1711 Also, is this blocking you? Just trying to understand the scope of the issue

[gcampbell-msft]

@FunkyGibbon1711 Also, one thing I'm not fully understanding, is that looking at the project you've zipped and provided us, I don't see any tests being added by your project, could you explain more about how you're adding tests?

[FunkyGibbon1711]

@gcampbell-msft This is not a blocker for me, we were able to just revert the CMakeTools extension version and it continued working. Here are some screenshot showcasing the issue (the test file is located in tester/src and added in tester/CMakeLists.txt:

The test configuration in tester/CMakeLists.txt - running the same test file with and without fixtures:

The test source file being ran:

The tests are all completing and passing as expected:

CTest marks the test with fixtures (helloyouWithFixture) as a failure as "Test results not found"

The issue occurs on my AWS Virtual Desktop running Amazon Linux 2 (CentOS effectively), but the project is configured to only run on x86 architecture.

[hippo91]

@FunkyGibbon1711 i tried to reproduce on the project you kindly provided but it appears that the archive is missing the tester directory. Is it normal?

[FunkyGibbon1711]

@hippo91 Apologies for this, here a new zip containing the relevant files:
testProject.zip

[gcampbell-msft]

@FunkyGibbon1711 So, to clarify, you're saying that in the output window, it says that the tests passed, but in the editor, it says that it failed due to "Test results not found"?

[FunkyGibbon1711]

@gcampbell-msft Yes that's the issue I'm seeing

[gcampbell-msft]

@FunkyGibbon1711 Ah, so it doesn't block any workflows, but it's very inconvenient for it to say that it's failing in one place even though it's not. Got it. We'll investigate this. I haven't been able to reproduce it yet, but I will work with our team to reproduce and get a fix in. Thanks!

[hippo91]

@FunkyGibbon1711 thanks for the new version of the archive.

However i am still unable to reproduce the behavior you described.

The screencast below shows what i tried:

• In the lib directory, i use CMake to clean, configure and build;
• In the tester directory, i use CMake to clean, configure and build;
• In the Test Explorer Panel i launch all the tests.

They all succeed and in the CMakeLists.txt file where tests are registered, they are marked with a green sign.

TestResultNotFound_2.mp4

Have you an idea to help us to reproduce the bug you are facing?

[Seb-Altwy]

Got the same issue here for a few days/weeks. (can't tell if it comes from a specific CMake Tools version though, as I've only started working more seriously on tests around those days anyway)

Multiple tests are passing fine (all marked as "passed" in the terminal when launched from a "ctest" command, or in the "Output" logs when triggered by the CMake tools "Run tests" button)... but the "Test results not found" error also pops up on the related CMakeLists "add_test(...)" line and on both "Test results" and "Testing" IDE tabs.

In my experience the key of the issue is the "FIXTURES_REQUIRED" property (via "set_tests_properties()" in CMakeLists). When not set, all is fine. When set, the error occurs despite all tests passing fine (and I'm sure it's correctly set in my case as it's showing the expected behavior on the tests running order). Wild guess: perhaps an issue on the tests results log files naming or read timing when having co-dependent tests?

[Seb-Altwy]

Really easy to test it with a minimalist project. (screenshots below)
Adding the "FIXTURES_REQUIRED" property triggers the issue (in addition to changing the tests order as expected when using the "ctest" command, but I don't show this very point on my screenshots).

Note: Tests order in the CMakeLists doesn't change anything in their execution nor the issue triggering, the issue seems to really come by the single fact that a "FIXTURES_REQUIRED" property is (correctly) set.

Side note: When hitting the "Run tests" CMake Tools' button all test are launched "one by one" (I mean for example test2 is launched itself, then test1 is launched aside with test2 once again because it's a prerequisite of test1 and no matter if it was launched already before that) instead of all in a single row as it would when using the "ctest" command (in that case all tests are only ran once each, only their order is changed to match the various "FIXTURES_REQUIRED" dependencies).

EDIT: Sorry, on the second screenshot you can't see it because the error message is actually hiding it, but I've just uncommented the "FIXTURES_REQUIRED" line to trigger the issue, and no other change. :)

[gcampbell-msft]

@FunkyGibbon1711 I have a PR here: #3823 that should fix the issue.

Could you install this vsix to confirm?
cmake-tools.zip

[FunkyGibbon1711]

@gcampbell-msft Will install this today and see if it resolves my issue. Thanks for the quick turnaround :)

[FunkyGibbon1711]

@gcampbell-msft I'm not sure if this is an error on my end however I can't seem to install this extension? I have unzipped the file, but there is only a .vsixmanifest file available which doesn't appear as a viable install file from the file selector opened by the Extensions: install from VSIX. I have tried installing on the command line using code --install-extension.vsixmanifest but this does not work either. Could you please provide some further insight?

[gcampbell-msft]

@FunkyGibbon1711 Ah, I should have provided more context, you'll have to change the file extension from .zip to .vsix.

So the process will be

1. Download the file.
2. Rename the extension from .zip to .vsix.
3. Install through the vscode extensions pane:

[FunkyGibbon1711]

@gcampbell-msft I can confirm I'm not longer getting the issue when using the vsix you have provided :)

[v-frankwang]

@FunkyGibbon1711 Thank you very much for your reply, waiting for this PR: #3823 this will be fixed after the merge.

[Ashley-Li]

Verified on CMake Tools v1.18.43(release), this issue fixed. Now there's no "Test Results Not Found" error appears.