limit parallel test execution OVER multiple assemblies

[schmitch]

currently we have a big solution that basically runs a shit ton of test suites, we basically have like 24 indepedent csproj's that all contain tests.
while we can probably reduce some of them, preferably we want them grouped. some of them even flip some appcontext that are than frozen for the rest of the assembly run.

is there any knob, preferably by adding a config file to limit how many csproj's can run in parallel?
I think the only possible solution would be to mark the csprojs that they should not run tests and create less grouped projects that are used for test discovery.

is that the only real solution or am I wrong? I tried with other test projects doing something, but I think that is even harder in nunit.
its also an extrem hussle that dotnet test has no hook whatsover that runs BEFORE ALL assemblies and AFTER ALL assemblies (i know that tunit (or others can't do anything against that))
Maybe I'm missing something and do not see the obvious solution for the problem tough.

[thomhurst]

@schmitch I believe if you run dotnet test against a solution, each test project only runs sequentially. I don't think they run in parallel?

[schmitch]

well it looks like it does:

whats even more strange is the huge amount of workers. over 80 workers for a single project?

[thomhurst]

Hmm interesting. Since running dotnet test against a solution will spawn separate processes for each project, this is outside of TUnit's control really. TUnit doesn't really have control over dotnet test command unfortunately - even less so when it's against a solution instead of a project.

As a workaround, you could create an 'orchestrator' project like MyNamespace.Tests.Orchestrator and add project references to all of your test projects within it. Then dotnet test that one orchestrator project, and then that should just be a single process, but still run tests from all referenced assemblies.

[schmitch]

yeah thats what I'm planing, unfortunatly do to an appswitch I can't group all into one (yet)

Hmm interesting. Since running dotnet test against a solution will spawn separate processes for each project, this is outside of TUnit's control really

thats what I tought. I also looked at MTP if there is anything about it, but unfortunatly, no. I do not understand why they do not try to fix that problem, because some stuff would be way easier if there would be mtp hooks, before all, after all projects.
(like report merging for example ;), test container management, etc, at the moment all these things needs to be a little bit hacky)

[muhammadmobi]

There is an SDK-level knob. Which one depends on how dotnet test runs your projects:

.NET 10 SDK with the Microsoft.Testing.Platform runner (global.json has "test": { "runner": "Microsoft.Testing.Platform" }): test modules run in parallel by default, up to Environment.ProcessorCount. Limit them with:

dotnet test MySolution.sln --max-parallel-test-modules 4

(docs). It's command-line only: the SDK reads no config file or environment variable for it (source), so put it in your script or CI step.

VSTest-based dotnet test (.NET 8/9 SDK, or no runner set in global.json): the projects are parallelised by MSBuild, so turn that off:

dotnet build MySolution.sln
dotnet test MySolution.sln --no-build -p:BuildInParallel=false

That's the recommendation from the SDK team for exactly this question. If you want it in a file, MSBuild picks up arguments from a Directory.Build.rsp in the repo root. That also affects every dotnet build there, though, so I'd keep it in the test script. (I haven't tested the .rsp route with dotnet test myself.)

Your ~80 workers per project are TUnit's in-process parallelism, which is separate. Cap that with --maximum-parallel-tests (docs).

And you're right that neither runner has before-all/after-all hooks across projects.

[schmitch]

That sounds like everybody needs to always set the flag..

[muhammadmobi]

For now, yes. With the MTP runner it's a per-invocation option only: the SDK reads --max-parallel-test-modules from the command line and otherwise uses Environment.ProcessorCount (option, where it's used). There's no global.json setting or environment variable for it.

So the practical options are:

• Put the command in one entry point that everyone and CI use (a test.sh / test.ps1, a Cake or Nuke target, or your CI template), so nobody types the flag by hand.
• For the projects that can share a process, use the orchestrator project @thomhurst suggested. Fewer modules means the limit matters less, and the ones that flip AppContext switches can stay separate.

If you want a repo-level setting, it's worth a feature request on dotnet/sdk. I couldn't find an existing one.