dotnet test seems to ignore runsettings MaxCpuCount

[flcdrg]

Description

dotnet test when run against a solution that contains multiple test projects, appears to spin up a parallel test runner for each test project.

Attempting to limit to one test runner at a time via the MaxCpuCount value in a .runsettings file doesn't work.

Steps to reproduce

1. Create a solution with two or more test projects that use NUnit
2. Create a .runsettings file that sets MaxCpuCount=1
3. run dotnet test --settings pathtofile.runsettings

Expected behavior

Tests assemblies are run sequentially

Actual behavior

Test assemblies are run in parallel (seen in my case where the tests are database tests and they end up fighting for access to the database)

Diagnostic logs

Environment

Windows 10 1809, vstest.console 15.0.28307.421
.NET Core SDK 2.2.200/2.2.202

[ShreyasRmsft]

@flcdrg vstest receives one dll at a time when you run dotnet test on an sln.

When you run dotnet test on an sln, after dotnet finishes build it passes the built binary on to vstest to run the tests. This means that the build is happening in parallel and soon after the tests are starting.

MaxCpuCount will have no impact on this process whatsoever. Since we only ever receive one dll. Your problem is that dotnet itself is creating these in parallel. MaxCpuCount is a vstest input and dotnet ignores this.

One way to get your dlls to run in sequence is to make sure build happens in sequence. Otherwise can you open an issue in the dotnet cli repo to requesting for a flag or parameter to ensure tests run in sequence regardless of when build completes.

[flcdrg]

And this is still the case if you run dotnet test --no-build --no-restore ?

[OsirisTerje]

@ShreyasRmsft It is unfortunate , that dotnet test doesn't use the information in the runsettings file. This file can be added as a parameter for dotnet test, and for the user it would feel natural that when one turns off parallel here, it also works for dotnet itself, and not just for the underlying vstest.

[ShreyasRmsft]

@OsirisTerje agreed. The current experience is a little unintuitive.

[mayankbansal018]

@flcdrg , @OsirisTerje dotnet test currently works on a single project, i.e. even you pass a solution file to it, internally dotnet test is invoked individually for every UT project. Doing this ensures that test doesn't have to wait till build is complete, & helps in performance. Hence there is no need for it to honor MaxCpuCount

However we are working on bringing in support of running dlls with dotnet test, the spec is yet to be closed, but the idea is to bring parity with how dotnet vstest works.

Current dotnet vstest honor's MaxCpuCount, & eventually in cases of dll, dotnet test will also start to do so.

[OsirisTerje]

Thanks @mayankbansal018 !
But, this doesn't seem to match the issue description on top. Each dll is run separately, but if they talk to a single database (integration test), you will have crashes, and then no way to control it. I think that is what the reporter says.

[Mpdreamz]

Even the fact that console loggers will be intermixed makes me want to run sequentially on CI to get sane output. MaxCpuCount is not a fix.

Please reconsider reopening this.

[papatelst]

I have Visual studio 2019 Version 16.5.0 Preview 2.0 and use .net core version "3.1.100"
Based on documentation at link, I tried using following command to restrict MaxCpuCount to 2 threads for my 1 test project called UITests, which has 3 tests ONLY with nunit category = foo.

vstest.console.exe artifacts\tests\UITests.dll /Settings:UITests\foo.runsettings /TestCaseFilter:"TestCategory=foo"

Under UITests project, I have following foo.runsettings xml file
<?xml version="1.0" encoding="utf-8" ?> <RunSettings> <RunConfiguration> <MaxCpuCount>2</MaxCpuCount> </RunConfiguration> </RunSettings>

However, it opens 3 threads in parallel, instead of 2., which basically means it ignores MaxCpuCount configuration.

I also tried dotnet.exe test command as follows which did not work either and opens 3 threads in parallel.
dotnet.exe test UITests\UITests.csproj -s UITests\foo.runsettings --filter "TestCategory=foo" --no-build

Please guide as to what am I doing wrong here.
Per comments above, it seems that it should atleast work with vstest.console.exe command if not with dotnet.exe test command.

I cannot use Nunit parameters as .net core tests cannot be run with nunit parameters because Nunit console does not work with .net core yet as mentioned at issue

My UITests project has assemblyinfo.cs file with following settings which seems to control the number of parallel threads and .runsettings file settings seem to not override this.

using NUnit.Framework;
//ParallelScope
[assembly: Parallelizable(ParallelScope.Fixtures)]
[assembly: LevelOfParallelism(16)]

Found the SOLUTION - https://github.com/nunit/docs/wiki/Tips-And-Tricks#NumberOfTestWorkers
This parameter called NumberOfTestWorkers in .runsettings file seems to override the LevelOfParallelism in assemblyinfo.cs in my .net core project.

[jefflill]

FYI: The NumberOfTestWorkers trick mentioned by @papatelst described in detail for xunit here:

https://xunit.net/docs/configuration-files#parallelizeAssembly

(edit: pasted the wrong link)

[smchan514]

For anyone still looking for a solution, there is the command line option -m:1 which specifies the maximum number of concurrent processes to use when building.

dotnet test -m:1 MyUnitTests.sln

Ref:

1. Add parameter to ensure tests run in sequence regardless of when build completes dotnet/sdk#10178
2. MSBuild command-line reference -- Switches

[aaronasmith]

For anyone still looking for a solution, there is the command line option -m:1 which specifies the maximum number of concurrent processes to use when building.

dotnet test -m:1 MyUnitTests.sln

Ref:

1. Add parameter to ensure tests run in sequence regardless of when build completes dotnet/sdk#10178
2. MSBuild command-line reference -- Switches

This seems to have been slightly broken in the latest SDK. The -m:1 switch must come directly after dotnet test and before the solution file or it is ignored.