-
Notifications
You must be signed in to change notification settings - Fork 431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't run CTest tests in parallel #3091
Comments
Thanks for the feedback! Marking this as a feature request for now based on this discussion. For others looking at this issue, please 👍 react to help us decide what to prioritize. |
Parallel execution of tests executed by CTest worked with previous version 1.13.45. It does not work with version 1.14.30. It seems this version runs each test one by one instead of letting CTest schedule the order of execution of tests and use parallel execution when possible. It is visible in logs of #3126 |
It seems to me this should be labelled "regression", then, instead of "feature request"? |
Parallel execution was working for me as well until I updated. I agree with @bilderbuchi that this seems like a regression. |
This is a pretty huge regression. Before the update I was able to run all my tests (> 300) under a second, now it takes over a minute... |
Was filing a new issue while noticed this one. It's a huge regression in my case, I have ~6.5k mostly simple google tests in 10 binaries, previously it was very fast to run them all. Now it takes around 30 minutes. Just checked, running a single debug binary with all ~650 tests took 2 seconds... |
Just ran into this as well. It seems that #3032 began running tests individually, one by one, using a for loop: vscode-cmake-tools/src/ctest.ts Lines 433 to 436 in 7879373
For an 8-core system, this is an 8x slowdown. I'm using the command line for now. |
This also seems related to #3149. |
@OleksandrKvl I apologize, I don't think it was released on the marketplace yet at that time. It should be available now though! |
When I press
but since it runs |
@OleksandrKvl this is after setting |
@snehara99 yes |
@OleksandrKvl Thanks for confirming! Could you by chance share an extended execution log? I want to see if it's different than what I see after the initial command execution line |
please clarify what do you mean by "extended execution log" |
exactly what you sent above, but with more lines included |
OK, here I started and immediately stopped test runner:
|
Thank you, I am able to repro the issue and I will have a fix out soon. In the meantime, invoking tests from the |
This is related to issue #3151 so I'm making a note here |
I for one am finding the Project Status View's interface a bit unintuitive: the "[All tests]" entry under "Tests" doesn't actually have any run options, and the "Play" button at the top appears to be equivalent to launching the selected target under the "launch" section. So it's very easy to click "All tests" and hit "play" to run them, resulting in it running something else entirely. Possibly a "Tests" button at the top would help. Personally, I don't tend to use Project Status View anyway - I want to run my tests off the Testing pane. If there's a one-click option to run all the tests via the CMake pane or the status bar, that's nice, but that's not primarily where I expect to be running the tests from - and I want them to work in parallel there too. Pretty please? :) |
@rjra100 Thank you for your feedback! We will take it into account as we are currently redesigning the Project Status View. It would be helpful if you could make a new issue with the request of running tests in parallel from the Test Explorer as it is a bit out of scope for this issue. We will take a look at it as soon as we can :) |
@OleksandrKvl If you upgrade to the newest pre-release channel, the |
@snehara99 thanks for the work done until now. I don't mean to sound ungrateful, but as I reported when I opened the issue, I usually run tests from the "Testing" pane, using the button "Run tests" with the double-play-like icon. Although running the tests by clicking on "Run CTest" in the status bar is now parallelized, running from the "Testing" pane isn't, and is still slow. Earlier you said this is a bit out of scope for this issue, but I'd like to disagree. It's exactly the original use case reported, and as such the present solution fixes a related issue, but not the exact same issue. I'm not sure how hard it would be to adapt this to work in the "Testing" pane. If you feel it would be better to open another issue specifically targeting this way of running the tests, let me know and I'll do it. |
@snehara99 thanks, now it works as you explained. In my case, 6.5k tests were executed in 37 seconds and I still wonder why it's so slow comparing to running a test binary by hand which takes only 2 seconds (the whole suite has 10 binaries). But I guess it's related to |
Executing the same |
FYI, since
but running
This seems to be a known problem in CTest and might be fixed in future releases. |
@OleksandrKvl Thanks for investigating the root of the issue. If a fix is released for CTest in the future, then we'll integrate it asap. If you like, you can create a new issue to track this problem moving forward. As for the original issue, the fix should now be available in the main release channel, so I am going to close this issue as completed. |
I am using CMake Tools 1.16.2 (Pre-release), and I use Is it supposed to work with this version? |
@snehara99 can you please advise with regards to my previous comment? As the original submitter, this doesn't fix the issue at all for me, due to the different path I take for running the tests (and which was reported as such when I opened the issue). Should I open a new issue |
Hi @dgazzoni, sorry for the late reply. I must have missed your comment while replying to others. I agree that your issue is still in scope with regards to the original post, however it would be very helpful for tracking on our end if you could create a new issue with this specific request to run parallel tests from the testing pane. Thank you for bumping this, we will start investigating as soon as the new issue is created! |
Hi @Zitrax could you please start a new issue so we can track this separately? I think this may be a slightly different issue than the one that was addressed here, and we will need to do more investigation. Thanks! |
Thanks. I just opened #3322 to cover this use case. |
Brief Issue Summary
Not sure if this is some project misconfiguration on my side, but CTest tests are not run in parallel. For every test
<test_name>
in my project, I see this in my output tab:While it actually goes through the motions of running in parallel (by passing
-j10
), because it filters the list of tests down to a single one (since<test_name>
is the name of one single, specific test), there is no opportunity to actually run in parallel.I get this behavior by running it from the "Testing" pane in the left-side of the VS Code screen: either clicking the button whose tooltip is "Run Tests" and runs all tests, clicking on the "play"-shaped button next to CTest or the same button next to my project folder under CTest. The behavior is also the same if I choose "CMake: Run Tests" or "Test: Run All Tests" from the command palette.
I can't see how this could be the intended behavior, as there's even a setting "Cmake > Ctest: Parallel Jobs" in the extension -- which BTW is set to 0 in my case, and so is "Cmake: Parallel Jobs"; indeed you can see above I have a
-j10
in my CTest command line. Could I get some pointers about what I might be doing wrong, perhaps a setting or something in my CMakeLists.txt that is blocking parallel tests? If by chance this is intended behavior, I'd like to turn this into a feature request to get tests to run in parallel as this would speed up my workflow a lot.CMake Tools Diagnostics
No response
Debug Log
No response
Additional Information
No response
The text was updated successfully, but these errors were encountered: