version 1.14 runs CTest "setup tasks" and "cleanup task" for each test instead of only once

[ebouteillon]

Brief Issue Summary

version 1.14 runs CTest "setup tasks" and "cleanup task" for each test instead of only once. This affects a lot the duration of running tests with fixtures.

Below a minimal sample of CMakeLists.txt to reproduce.

CMAKE_MINIMUM_REQUIRED (VERSION 3.7)

project(cmake_fixture)

enable_testing()

# Create a fixture to setup before tests / cleanup after tests
add_test(NAME pre_ctest COMMAND sleep 5)
add_test(NAME post_ctest COMMAND sleep 1)
set_tests_properties(pre_ctest PROPERTIES FIXTURES_SETUP database)
set_tests_properties(post_ctest PROPERTIES FIXTURES_CLEANUP database)
set_tests_properties(pre_ctest post_ctest PROPERTIES RESOURCE_LOCK database_access)

add_test(NAME test_1 COMMAND sleep 1)
set_tests_properties(test_1 PROPERTIES FIXTURES_REQUIRED database)
set_tests_properties(test_1 PROPERTIES RESOURCE_LOCK database_access)

add_test(NAME test_2 COMMAND sleep 1)
set_tests_properties(test_2 PROPERTIES FIXTURES_REQUIRED database)
set_tests_properties(test_2 PROPERTIES RESOURCE_LOCK database_access)

add_test(NAME test_3 COMMAND sleep 1)
set_tests_properties(test_3 PROPERTIES FIXTURES_REQUIRED database)
set_tests_properties(test_3 PROPERTIES RESOURCE_LOCK database_access)

With version 1.13.45 : duration 9 seconds, pre_ctest and post_ctest are run only once

[main] Building folder: buggy_cmake 
[build] Starting build
[proc] Executing command: /opt/cmake/bin/cmake --build /home/axis/workspace/buggy_cmake/build --config Debug --target all -j 14 --
[build] Build finished with exit code 0
[proc] Executing command: /opt/cmake/bin/ctest -j14 -C Debug -T test --output-on-failure
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project /home/axis/workspace/buggy_cmake/build
[ctest]     Start 1: pre_ctest
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest] 1/5 Test #1: pre_ctest ........................   Passed    5.00 sec
[ctest]     Start 4: test_2
[ctest] 2/5 Test #4: test_2 ...........................   Passed    1.00 sec
[ctest]     Start 3: test_1
[ctest] 3/5 Test #3: test_1 ...........................   Passed    1.00 sec
[ctest]     Start 5: test_3
[ctest] 4/5 Test #5: test_3 ...........................   Passed    1.00 sec
[ctest]     Start 2: post_ctest
[ctest] 5/5 Test #2: post_ctest .......................   Passed    1.00 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 5
[ctest] 
[ctest] Total Test time (real) =   9.01 sec
[ctest] CTest finished with return code 0

With version 1.14.30 : duration 27 seconds
pre_ctest and post_ctest are run 4 times

[main] Building folder: buggy_cmake 
[build] Starting build
[proc] Executing command: /opt/cmake/bin/cmake --build /home/axis/workspace/buggy_cmake/build --config Debug --target all -j 14 --
[driver] Build completed: 00:00:00.029
[build] Build finished with exit code 0
[proc] Executing command: /opt/cmake/bin/ctest -j14 -C Debug -T test --output-on-failure -R ^pre_ctest$
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project /home/axis/workspace/buggy_cmake/build
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest]     Start 1: pre_ctest
[ctest] 1/1 Test #1: pre_ctest ........................   Passed    5.00 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   5.00 sec
[ctest] CTest finished with return code 0
[proc] Executing command: /opt/cmake/bin/ctest -j14 -C Debug -T test --output-on-failure -R ^post_ctest$
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project /home/axis/workspace/buggy_cmake/build
[ctest]     Start 2: post_ctest
[ctest] 1/1 Test #2: post_ctest .......................   Passed    1.00 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   1.00 sec
[ctest] CTest finished with return code 0
[proc] Executing command: /opt/cmake/bin/ctest -j14 -C Debug -T test --output-on-failure -R ^test_1$
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project /home/axis/workspace/buggy_cmake/build
[ctest]     Start 1: pre_ctest
[ctest] 1/3 Test #1: pre_ctest ........................   Passed    5.00 sec
[ctest]     Start 3: test_1
[ctest] 2/3 Test #3: test_1 ...........................   Passed    1.00 sec
[ctest]     Start 2: post_ctest
[ctest] 3/3 Test #2: post_ctest .......................   Passed    1.00 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 3
[ctest] 
[ctest] Total Test time (real) =   7.01 sec
[ctest] CTest finished with return code 0
[proc] Executing command: /opt/cmake/bin/ctest -j14 -C Debug -T test --output-on-failure -R ^test_2$
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project /home/axis/workspace/buggy_cmake/build
[ctest]     Start 1: pre_ctest
[ctest] 1/3 Test #1: pre_ctest ........................   Passed    5.00 sec
[ctest]     Start 4: test_2
[ctest] 2/3 Test #4: test_2 ...........................   Passed    1.00 sec
[ctest]     Start 2: post_ctest
[ctest] 3/3 Test #2: post_ctest .......................   Passed    1.00 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 3
[ctest] 
[ctest] Total Test time (real) =   7.01 sec
[ctest] CTest finished with return code 0
[proc] Executing command: /opt/cmake/bin/ctest -j14 -C Debug -T test --output-on-failure -R ^test_3$
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest] Cannot find file: /home/axis/workspace/buggy_cmake/build/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Test project /home/axis/workspace/buggy_cmake/build
[ctest]     Start 1: pre_ctest
[ctest] 1/3 Test #1: pre_ctest ........................   Passed    5.00 sec
[ctest]     Start 5: test_3
[ctest] 2/3 Test #5: test_3 ...........................   Passed    1.00 sec
[ctest]     Start 2: post_ctest
[ctest] 3/3 Test #2: post_ctest .......................   Passed    1.00 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 3
[ctest] 
[ctest] Total Test time (real) =   7.01 sec
[ctest] CTest finished with return code 0

CMake Tools Diagnostics

{
  "os": "linux",
  "vscodeVersion": "1.77.3",
  "cmtVersion": "1.14.30",
  "configurations": [
    {
      "folder": "/home/axis/workspace/buggy_cmake",
      "cmakeVersion": "3.26.3",
      "configured": true,
      "generator": "Unix Makefiles",
      "usesPresets": false,
      "compilers": {
        "C": "/opt/rh/devtoolset-8/root/usr/bin/gcc",
        "CXX": "/opt/rh/devtoolset-8/root/usr/bin/g++"
      }
    }
  ],
  "cpptoolsIntegration": {
    "isReady": true,
    "hasCodeModel": false,
    "activeBuildType": "Debug",
    "buildTypesSeen": [
      "Debug"
    ],
    "requests": [],
    "responses": [],
    "partialMatches": [],
    "targetCount": 0,
    "executablesCount": 0,
    "librariesCount": 0,
    "targets": []
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": true
    }
  ]
}

Debug Log

No response

Additional Information

No response

[xisui-MSFT]

Looks like regex didn't work in ctest command... I'll try to repro.

[xisui-MSFT]

Ah I see it's a pre-test + post-test fixture. This is then a duplicate of #3091. Please "thumb up" in #3091 so we can prioritize this feature.

Closing this ticket as duplicate.