Independent CTest invocations break RESOURCE_LOCK and test fixtures

[MathiasMagnus]

I'm developing a local CI runner implemented as a CMake module to overcome the limitations of GitHub and Gitlab which don't cater to local development by enabling running CI pipelines locally. (They were not designed for this use case.)

My module in development accepts a test matrix definition and executes them locally inside docker containers. Containers spin up as CTest fixture setups (they were designed for such use cases) and are discarded as fixture cleanups. the module also allows uses of RESOURCE_LOCK to forbid tests running in parallel when using a shared resource. (Such is the actual case of multi-configuration build systems, MSBuild or Ninja Multi-Config generators for eg. which do not tolerate parallel invocations of differing configurations (Build and Release) on the same build tree concurrently.)

The problem with the extensions current design is that both fixtures and resource locks only work as part of a single CTest invocation. When each test is run in a separate CTest invocation, multiple instances of the same resource lock and fixture may be in flight, which results in breaking compilations and random errors.

I'm not sure if it's possible with the current Test Explorer API, but I would like to ask for no more than one CTest invocation when running all (or any subset) of the tests.

[MathiasMagnus]

This is the UI of a multi-root workspace, where one folder is the root of the library being developed and the other is a subfolder of the project with the Dockerfiles of the images and the scripts to execute the CI pipeline. This auxiliary project has build steps for building the images from the Dockerfiles and each test is a step of the CI pipeline of one test matrix element. Dependencies are properly set among tests to form parallel pipelines.

I would like to press the "Run tests" button and see the green lights light up as they would when executing CTest on the command-line, which currently works as intended.

[fredericbonnet]

Indeed, the extension runs each test in isolation, and if parallel execution is enabled this means that tests with resource locks would still run in parallel despite the lock. As a quick workaround you can disable parallel execution on your project (set the cmakeExplorer.parallelJobs settings property to 1 or negative value).

An ideal way to respect all of CTest constraints would be to use its native parallel execution option -j/--parallel to run all jobs instead of letting the extension schedule the runs itself, alas you couldn't monitor the progress of each individual tests because CTest standard output is not machine-readable. So I consider either disabling the parallel execution of tests with resource groups/locks, or grouping them together in one CTest run. Unless I manage to use the native parallel execution of course.

[MathiasMagnus]

Thanks @fredericbonnet for the reply.

In theory, that hotfix will make execution correct, but it will make my tests roughly 32X slower. Not because I have 32 cores, but because tests not only use resource locks, but also fixtures and dependencies.

0. Fixture setup: spin up container X.
1. Configure project inside contiainer
2. Build project inside container (depends on 1.)
3. Run unit tests inside container (depends on 2.)
4. Install in container (depends on 3.)
5. Fixture cleanup: discard container

Separate CTest invocations means running test 1. requires running: 0+1+5. Running test 2 will run: 0+1+2+5 and so on and so forth.

0. Runs 4 times
1. Runs 4 times
2. Runs 3 times
3. Runs 2 times
4. Runs 1 times
5. Runs 4 times

So running 4 tests and 2 fixtures becomes 10 tests and 8 fixtures. Instead of the current 270 test cases on 16 cores, I'll be executing roughly 500 tests, all in serial. Execution will be like the song: The 12 days of Christmas.

So yes, the workaround in theory works, but in practice renders the UI useless (in this niche use case). The point of this is to be able to run rapid CI testing locally. This serial workaround will work if I don't run the entire test suite, but only 1-2 CI matrix elements final test case which is suspect to failure.

[fredericbonnet]

I've rewritten the test scheduling code to use the ctest command directly instead of letting the extension do the heavy lifting. This should fix the issue. Parsing the output proved easier than expected, though it required a bit of refactoring on the extension side. Can you install the attached package to check that it works for all your use cases? (rename .zip to .vsix beforehand)

cmake-test-adapter-0.12.0.zip

[MathiasMagnus]

Fantastic! Works like a charm! The extension is becoming awesomer by the day!

ps.: what materials did you read to pick up VS Code extension development? I really want to learn VS Code extension authoring, and fixing the deprecation warnings upon installing the extension may be a good starting point.

code --install-extension .\cmake-test-adapter-0.12.0.vsix
Installing extensions...
(node:20024) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Extension 'cmake-test-adapter-0.12.0.vsix' was successfully installed.

I really want to chime in developing CMake Tets Explorer, I've fiddled with the extension tutorial of VS Code, but the jump from C/C++ (and it's tooling) to TypeScript/NPM is daunting.

[fredericbonnet]

For this extension I based my work on the Test Explorer example, so a lot of scaffolding was already done: https://github.com/hbenl/vscode-example-test-adapter

This is the first and only VS code extension that I have authored so I can't really talk about a fresh start, however the MS docs have good resources and example code:

https://github.com/microsoft/vscode-extension-samples

Perhaps the best way is to start from an existing extension that is close to what you want to achieve, and build up from there.