Issues when running tests in parallel

[Tymon13]

Brief Issue Summary

This is sort of follow-up to #3091. I have two, possibly related issues:

Tests do not follow the philosophy "Visual Studio Code settings do not change preset behaviour"

The doc states

To invoke CTest, run CMake: Run Tests from the command palette. This is the same as running ctest --preset <testPreset> from the command line, where <testPreset> is the name of the active Test Preset.

and

CMakePresets.json should be the source of truth for all settings related to configure, build, and test. This eliminates behavior specific to Visual Studio Code and ensures that your CMake and CTest invocations can be reproduced from the command line.

Neither of these is true unless user has specifically set "cmake.ctest.allowParallelJobs": true, in settings (defaults to false). If this setting is not on, CMake: Run Tests does the following:

for test in $detected_tests
do
    ctest --preset <testPreset> -R $test
done

While this does pass --parallel N if preset has "execution": {"jobs": N}, that option does nothing since there is only one test executing due to -R effect.
I don't know what the correct action is. Preferably, the behaviour should stick to the philosophy of "settings do not interfere with presets" and either the setting should be true by default or removed altogether (since it does affect running tests outside of CMakePresets.json).

CMake: Run Tests and Tests: Run All Tests have different behaviour

Once I do set "cmake.ctest.allowParallelJobs": true,, CMake: Run Tests and clicking the flask in CMake tab does execute tests in parallel as expected, but Tests: Run All Tests and clicking "Run Tests" in "Testing" tab does not execute tests in parallel, as if ignoring cmake.ctest.allowParallelJobs completely.
While not a critical issue, this prevents use of Testing tab and access to nice visual test results.

I didn't include Log diagnostics nor debug logs, since they don't seem very useful and I'd have to strip them of protected information first. If you need them, I can provide it to you.

CMake Tools Diagnostics

No response

Debug Log

No response

Additional Information

No response

[gcampbell-msft]

@Tymon13 Thanks for posting the issue. It sounds like, at the very least, we need to make sure that our docs are up-to-date with our actual implementation. Additionally, we need to continue investigating our testing implementation.

@xisui-MSFT FYI.

[gcampbell-msft]

@Tymon13 Could you update to the most recent pre-release version and confirm whether these findings are still accurate? We recently merged #3577 and I'm curious about what affects it has on this issue. Thanks.

[gcampbell-msft]

Either way, we have multiple reasons for running the tests like we do. Specifically, without the allowParallelJobs setting enabled, we want to run each test individually so that the Test Explorer can correctly track progress (assuming the test explorer integration is turned on). With the allowParallelJobs setting enabled, we run in parallel, but the Test Explorer may not track progress as well.

With these things in mind, I think this is simply a documentation bug, but I'd like to get your feedback in order to confirm this.