Parametrized testcases not running in parallel

[masaeedu]

Issue

I am trying to run several cases of a parametrized test in parallel. Here is the code I'm using:

class Dummy
{
    static TestCaseData Case(int i)
        => new TestCaseData(TimeSpan.FromSeconds(2)).SetName($"Case {i}");

    public static IEnumerable<TestCaseData> Cases()
        => Enumerable.Range(1, 5).Select(Case);

    [TestCaseSource(nameof(Cases)), Parallelizable]
    public void ItShouldSleep(TimeSpan t)
        => Thread.Sleep(t);
}

When I run this using the Visual Studio test runner, I can see the individual cases being executed one at a time, and the overall test run takes ~10 seconds. I would expect all the cases to be executed simultaneously on separate threads, and for the overall run to take ~2 seconds.

Setup

I am using the following .NET Standard based csproj file alongside the code above:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Library</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <ApplicationIcon />
    <StartupObject />
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
    <PackageReference Include="NUnit" Version="3.8.1" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.8.0" />
  </ItemGroup>

</Project>

NuGet packages:

• Microsoft.NET.Test.Sdk: v15.3.0
• Microsoft.NETCore.App: v2.0.0
• NUnit: v3.8.1
• NUnit3TestAdapter: v3.8.0

Visual Studio:

• NUnit 3 Test Adapter: 3.8.0.0
• Visual Studio: 15.3.5

[CharliePoole]

See https://github.com/nunit/docs/wiki/Framework-Parallel-Test-Execution

"Parallel execution is supported by the NUnit framework on desktop .NET runtimes. It is not supported in our Portable or .NET Standard builds at this time, although the attributes are recognized without error in order to allow use in projects that build against multiple targets."

IOW, parallel execution is not supported in your .NET Core test assembly.

@nunit/framework-team The above quote is at the very bottom of a usage note that many users will never read, so I think this is a bit of a documentation issue. The warning should be moved up in the usage not and added to the doc page for the attribute itself.

[masaeedu]

@CharliePoole Gotcha, thanks. Is there an issue tracking .NET Standard support for parallelism? Is the code just ifdefed out?

[masaeedu]

@CharliePoole Also, changing to <TargetFramework>net461</TargetFramework> doesn't seem to enable parallelism for me.

[CharliePoole]

@nunit/framework-team I updated the usage note and the descriptions of all three parallelism-related attributes in the docs.

@masaeedu Check to see which build of the framework is referenced in your test. If it's .NET Standard or (for older NUnit) Portable, then you get no parallelism. This is not exactly an issue for us to track. We can't support parallelism in either the .NET standard 1.3 or 1.6 build because they don't provide the support we would need for it. We have an issue for creation of a .NET standard 2.0 build, which would support it.

[masaeedu]

@CharliePoole I've gone down to <TargetFramework>net45</TargetFramework> and I'm still not getting any parallelism.

Here's the DLL that my NUnit package dependency is resolving to (you can see the metadata indicates it is the full framework build):

A full repro project is here: https://github.com/masaeedu/nunit-issue

[masaeedu]

Issue here was that I wasn't applying the attribute correctly. You need to have [Parallelizable(ParallelScope.Children)] on the containing class.

[ChrisMaddock]

I would have through Parallelizable(ParallelScope.Children) on the TestCaseSource method would have worked - as the method is technically a 'test-suite'. @CharliePoole - is this a bug, or have a mis-understood the API?

@masaeedu's code is a little different here to what he posted in Gitter - the example I was looking at was this:

    [TestFixture]
    class Deserialization
    {
        public static IEnumerable<TimeSpan> ShouldDeserializeAllCases() 
              => Enumerable.Repeat(0, 5).Select(x => TimeSpan.FromSeconds(2));

        [TestCaseSource("ShouldDeserializeAllCases"), Parallelizable(ParallelScope.Children)]
        public void ShouldDeserializeAll(TimeSpan t)
        {
            Thread.Sleep(t);
            Assert.AreEqual(1, 1);
        }
    }

[masaeedu]

@ChrisMaddock I've tried both those versions and neither works. The Parallelizable(ParallelScope.Children) is on the test method itself though, and not on the method that is the testcase source. Maybe that's what the issue is?

[ChrisMaddock]

No - I don't think putting the parallelizable attribute on the testcasesource method itself would do anything at all.

As you pointed out - the docomments do say ParallelScope.Children can't be used on methods - so the behaviour your seeing may be as expected - but it's surprising to me!

[masaeedu]

Perhaps there should be some runtime validation of attributes when nUnit reflects over the test assembly, and an error raised if nonsensical usage is detected.