Tests in parallel fixtures not running in parallel per namespace

[madelson]

I have a test project with several namespaces. I would like each namespace to run in parallel with other namespaces, but for the tests within each namespace to run serially.

My understanding from reading the docs is that this can be achieved with SetUpFixture. I've set up my project like so:

In the Namespace1 folder:

using NUnit.Framework;

namespace Namespace1;

[SetUpFixture, Parallelizable]
public class Namespace1SetUpFixture
{
}

public class Test
{
    [Test]
    public async Task SlowTest()
    {
        Console.WriteLine($"Starting at {DateTime.Now}");
        await Task.Delay(TimeSpan.FromSeconds(10));
        Console.WriteLine($"Finishing at {DateTime.Now}");
    }
}

In the Namespace2 folder:

using NUnit.Framework;

namespace Namespace2;

[SetUpFixture, Parallelizable]
public class Namespace2SetUpFixture
{
}

public class Test
{
    [Test]
    public async Task SlowTest()
    {
        Console.WriteLine($"Starting at {DateTime.Now}");
        await Task.Delay(TimeSpan.FromSeconds(10));
        Console.WriteLine($"Finishing at {DateTime.Now}");
    }
}

When I select and run Namespace1.Test and Namespace2.Test in the VS test explorer however, they run in serial. The console print statements show no overlap in the running times.

Am I doing something wrong or could this be an issue with NUnit or the VS adapter?

[rprouse]

I think this is a bug with Visual Studio and running individual tests. I expect your tests will run in parallel if you run all tests or if you run from the console.

[madelson]

@rprouse is this a known VS bug or just a guess? Worth filing something with VS itself? Or maybe this is an issue with the NUnit adapter?

[rprouse]

@OsirisTerje can you comment on this? Have you seen this before in VS?

[OsirisTerje]

I wasn't even aware that you could run namespaces in parallel. It is not in the docs, https://docs.nunit.org/articles/nunit/writing-tests/attributes/parallelizable.html

And, if I try it with VS it doesn't, but also when I try this with the NUnit3-console, it doesn't there either.

@madelson , how did you manage to run these in parallel?
And, where in the docs did you find this method documented?

[madelson]

@OsirisTerje I did not successfully get the namespaces to run in parallel, hence my post ;-).

I wonder if there is a better pattern for doing this? For example, I could set ParallelScope.Fixtures on the assembly level and then use an assembly-level action attribute to manually lock based on the test's namespace or similar.

[OsirisTerje]

If you move the namespaces into separate test projects, they will run in parallel, then it is the testhost doing that.
ParallelScope.Fixtures will do the job, but then that will apply to all fixtures. Is that a problem? Tests should be independent.
If you really need them to be sequential, the strategy you mention may work.

[madelson]

My tests are independent, but each namespace uses a different external resource (local database server) and I find that running in parallel within the namespace results in a higher rate of timeouts and transient failures.

[OsirisTerje]

@mandelson Sorry for not responding further (got lost in time. Afaik you can't run namespaces in parallel. You have to split into different projects, that would do it.

My understanding from reading the docs is that this can be achieved with SetUpFixture.

Where did you see this in the documentation?

@rprouse Would you like to add the possibility if at all possible, to run namespaces in parallel? If not doable, then this issue should be closed.

[madelson]

Where did you see this in the documentation?

@OsirisTerje sorry for the delayed response. I was looking at this page.

It says "Parallel SetUpFixture with non-parallel test fixtures: The group runs in parallel with other fixtures and groups. Within the group, only one fixture at a time may execute."

I read this as describing the behavior I want. Do you agree?

[OsirisTerje]

I agree that looks promising for what you want to achieve. But the fixtures then need to be set to NonParallelizable.

To check this out you can write a small repro, you can add that to the brand new nunit.issues repo. We can then try to work with it together to see if there is something you need to do in the code, or if the documentation is wrong.

[OsirisTerje]

@madelson Did you get any further with this?

[madelson]

@OsirisTerje sorry I've been busy and haven't gotten back to this. I still plan to if I can.

[Bartleby2718]

@madelson I think this is because you didn't have [Parallelizable] for public class Test. I tried to reproduce this in Bartleby2718/nunit-parallel-execution@e059d1b and got the following output:

========== Starting test run ==========
NUnit Adapter 4.5.0.0: Test execution started
Running selected tests in C:\Users\bartl\Documents\GitHub\nunit-parallel-execution\MyProject.Tests\bin\Debug\net8.0\MyProject.Tests.dll
   NUnit3TestExecutor discovered 4 of 4 NUnit test cases using Current Discovery mode, Non-Explicit run
N1OneTimeSetUp: Starting at 3/31/2024 9:19:40 PM
N2OneTimeSetUp: Starting at 3/31/2024 9:19:40 PM
N2OneTimeSetUp: Finishing at 3/31/2024 9:19:50 PM
N1OneTimeSetUp: Finishing at 3/31/2024 9:19:50 PM
N2SlowTest1: Starting at 3/31/2024 9:19:50 PM
N1SlowTest1: Starting at 3/31/2024 9:19:50 PM
N2SlowTest1: Finishing at 3/31/2024 9:20:00 PM
N1SlowTest1: Finishing at 3/31/2024 9:20:00 PM
N1SlowTest2: Starting at 3/31/2024 9:20:00 PM
N1SlowTest2: Finishing at 3/31/2024 9:20:10 PM
NUnit Adapter 4.5.0.0: Test execution complete
========== Test run finished: 4 Tests (4 Passed, 0 Failed, 0 Skipped) run in 30.2 sec ==========

which shows that:

1. OneTimeSetUps in different SetUpFixtures can run in parallel if they're decorated with Parallelizable
2. Tests in different namespaces, in this case N1SlowTest1 and N2SlowTest1, can run in parallel.
3. With [Parallelizable] (which is the same as [Parallelizable(ParallelScope.Self)]), tests within the same fixture run serially.

so I think the documentation is correct?