SetupFixture without namespace will make assembly-level Parallelable attribute useless

[kfstorm]

I've tried to combine parallel testing and SetupFixture. But no luck.

I added [assembly:Parallelizable(ParallelScope.Fixtures)] in code. And if I put SetupFixture into root namespace, then test fixtures in this assembly will run sequentially. However, if I put SetupFixture into the same namespace as other text fixtures, test fixtures can run in parallel.

I've tried both NUnit 3.4.1 and 3.5.0. The results are the same.

Below is a simple file to repro the issue.

using System.Threading;
using NUnit.Framework;

[assembly:Parallelizable(ParallelScope.Fixtures)]

[SetUpFixture]
public class GlobalSetup
{
    [OneTimeSetUp]
    public void Setup()
    {
        Thread.Sleep(5000);
    }
}

namespace NUnitTest
{
    [TestFixture]
    public class TestFixture1
    {
        [Test]
        public void Test1()
        {
            Thread.Sleep(5000);
        }
    }

    [TestFixture]
    public class TestFixture2
    {
        [Test]
        public void Test2()
        {
            Thread.Sleep(5000);
        }
    }
}

Here is the test result file.

<?xml version="1.0" encoding="utf-8"?>
<test-run id="2" name="NUnitTest.dll" fullname="GlobalSetup" testcasecount="2" result="Passed" start-time="2016-11-18 14:25:20Z" end-time="2016-11-18 14:25:35Z" duration="15.037411" total="2" passed="2" failed="0" inconclusive="0" skipped="0" asserts="0" random-seed="1862713937">
  <command-line><![CDATA[XXX\NUnit-3.5.0\bin\net-4.5\nunitlite-runner.exe  NUnitTest.dll]]></command-line>
  <filter />
  <test-suite type="SetUpFixture" id="1000" name="NUnitTest.dll" fullname="GlobalSetup" classname="GlobalSetup" runstate="Runnable" testcasecount="2" result="Passed" start-time="2016-11-18 14:25:20Z" end-time="2016-11-18 14:25:35Z" duration="15.037411" total="2" passed="2" failed="0" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.5.0.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.2.9200.0" platform="Win32NT" cwd="XXX\NUnitTest\NUnitTest\bin\Debug" machine-name="XXX" user="XXX" user-domain="XXX" culture="en-US" uiculture="en-US" os-architecture="x86" />
    <settings>
      <setting name="WorkDirectory" value="XXX\NUnitTest\NUnitTest\bin\Debug" />
      <setting name="NumberOfTestWorkers" value="8" />
    </settings>
    <properties>
      <property name="ParallelScope" value="Fixtures" />
      <property name="_PID" value="8328" />
      <property name="_APPDOMAIN" value="nunitlite-runner.exe" />
    </properties>
    <test-suite type="TestSuite" id="1006" name="NUnitTest" fullname="NUnitTest" runstate="Runnable" testcasecount="2" result="Passed" start-time="2016-11-18 14:25:25Z" end-time="2016-11-18 14:25:35Z" duration="10.023981" total="2" passed="2" failed="0" inconclusive="0" skipped="0" asserts="0">
      <test-suite type="TestFixture" id="1001" name="TestFixture1" fullname="NUnitTest.TestFixture1" classname="NUnitTest.TestFixture1" runstate="Runnable" testcasecount="1" result="Passed" start-time="2016-11-18 14:25:25Z" end-time="2016-11-18 14:25:30Z" duration="5.011831" total="1" passed="1" failed="0" inconclusive="0" skipped="0" asserts="0">
        <test-case id="1002" name="Test1" fullname="NUnitTest.TestFixture1.Test1" methodname="Test1" classname="NUnitTest.TestFixture1" runstate="Runnable" seed="1420857753" result="Passed" start-time="2016-11-18 14:25:25Z" end-time="2016-11-18 14:25:30Z" duration="5.003950" asserts="0" />
      </test-suite>
      <test-suite type="TestFixture" id="1003" name="TestFixture2" fullname="NUnitTest.TestFixture2" classname="NUnitTest.TestFixture2" runstate="Runnable" testcasecount="1" result="Passed" start-time="2016-11-18 14:25:30Z" end-time="2016-11-18 14:25:35Z" duration="5.004199" total="1" passed="1" failed="0" inconclusive="0" skipped="0" asserts="0">
        <test-case id="1004" name="Test2" fullname="NUnitTest.TestFixture2.Test2" methodname="Test2" classname="NUnitTest.TestFixture2" runstate="Runnable" seed="2116975853" result="Passed" start-time="2016-11-18 14:25:30Z" end-time="2016-11-18 14:25:35Z" duration="5.000674" asserts="0" />
      </test-suite>
    </test-suite>
  </test-suite>
</test-run>

[CharliePoole]

Note that this is in nunitlite. Should be same under console but we should confirm both.

[kfstorm]

@CharliePoole Here is the test result under console. It took 15 seconds too.

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-run id="2" testcasecount="2" result="Passed" total="2" passed="2" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.5.0.0" clr-version="4.0.30319.42000" start-time="2016-12-01 17:36:06Z" end-time="2016-12-01 17:36:21Z" duration="15.141362">
  <command-line><![CDATA[nunit3-console.exe  "C:\XXX\ClassLibrary1.dll"]]></command-line>
  <test-suite type="SetUpFixture" id="0-1000" name="ClassLibrary1.dll" fullname="GlobalSetup" classname="GlobalSetup" runstate="Runnable" testcasecount="2" result="Passed" start-time="2016-12-01 17:36:06Z" end-time="2016-12-01 17:36:21Z" duration="15.098554" total="2" passed="2" failed="0" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.5.0.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 10.0.14393.0" platform="Win32NT" cwd="C:\NUnit.3.5.0" machine-name="XXX" user="XXX" user-domain="XXX" culture="zh-CN" uiculture="zh-CN" os-architecture="x64" />
    <settings>
      <setting name="DisposeRunners" value="True" />
      <setting name="WorkDirectory" value="C:\NUnit.3.5.0" />
      <setting name="ImageRuntimeVersion" value="4.0.30319" />
      <setting name="ImageTargetFrameworkName" value=".NETFramework,Version=v4.5" />
      <setting name="ImageRequiresX86" value="False" />
      <setting name="ImageRequiresDefaultAppDomainAssemblyResolver" value="False" />
      <setting name="NumberOfTestWorkers" value="8" />
    </settings>
    <properties>
      <property name="ParallelScope" value="Fixtures" />
      <property name="_PID" value="8880" />
      <property name="_APPDOMAIN" value="test-domain-" />
    </properties>
    <test-suite type="TestSuite" id="0-1006" name="NUnitTest" fullname="NUnitTest" runstate="Runnable" testcasecount="2" result="Passed" start-time="2016-12-01 17:36:11Z" end-time="2016-12-01 17:36:21Z" duration="10.047220" total="2" passed="2" failed="0" inconclusive="0" skipped="0" asserts="0">
      <test-suite type="TestFixture" id="0-1001" name="TestFixture1" fullname="NUnitTest.TestFixture1" classname="NUnitTest.TestFixture1" runstate="Runnable" testcasecount="1" result="Passed" start-time="2016-12-01 17:36:11Z" end-time="2016-12-01 17:36:16Z" duration="5.034269" total="1" passed="1" failed="0" inconclusive="0" skipped="0" asserts="0">
        <test-case id="0-1002" name="Test1" fullname="NUnitTest.TestFixture1.Test1" methodname="Test1" classname="NUnitTest.TestFixture1" runstate="Runnable" seed="1350862790" result="Passed" start-time="2016-12-01 17:36:11Z" end-time="2016-12-01 17:36:16Z" duration="5.015906" asserts="0" />
      </test-suite>
      <test-suite type="TestFixture" id="0-1003" name="TestFixture2" fullname="NUnitTest.TestFixture2" classname="NUnitTest.TestFixture2" runstate="Runnable" testcasecount="1" result="Passed" start-time="2016-12-01 17:36:16Z" end-time="2016-12-01 17:36:21Z" duration="5.004419" total="1" passed="1" failed="0" inconclusive="0" skipped="0" asserts="0">
        <test-case id="0-1004" name="Test2" fullname="NUnitTest.TestFixture2.Test2" methodname="Test2" classname="NUnitTest.TestFixture2" runstate="Runnable" seed="1771661709" result="Passed" start-time="2016-12-01 17:36:16Z" end-time="2016-12-01 17:36:21Z" duration="5.000401" asserts="0" />
      </test-suite>
    </test-suite>
  </test-suite>
</test-run>

[CharliePoole]

@nunit/contributors Can somebody confirm this?

[Sebazzz]

Yes, developing my ordered testing library I found that IApplyToContext does not run when a SetupFixture is defined. I assumed it was well defined behavior.

The workaround is to apply the attribute to the SetUpFixture instead.

[CharliePoole]

Thanks @Sebazzz - the result is actually unintended. What happens is that the global SetUpFixture actually takes the place of the assembly in the hierarchy of tests. In fact, it should actually honor any attributes on the fixture plus any assembly-level attributes.

[kfstorm]

Guys, do you have a timeline? How long I need to wait until the fix is released?

[CharliePoole]

@kfstorm You can always see the status of any issue if you look at it online in GitHub. In this case, the issue is in our Backlog, meaning it's available for somebody to work on it. I put a comment on it as a result of my triage, so that folks would understand what has to be fixed.

Up to now, no team member or contributor has self-assigned the issue, so it's not likely it will be in the 12/31 release. Self-assignment is how we handle issues that are low or normal priority.

I can guess this is not a really satisfactory answer for you, but it's the best I have. 😞

[kfstorm]

@CharliePoole OK. Thanks for your explain.