NUnit3 sometimes hangs if SetUpFixtures are run in parallel

[RobinsonWM]

If I have multiple SetUpFixtures in the same namespace, and I enable parallelization at the Assembly level, NUnit 3.0.1 sometimes hangs after running the [OneTimeSetUp] methods, before running the tests.

It happens with both the console runner and the Visual Studio test adapter. In my case it remains hung for 24 hours (at which point our automated build tool kills the process.)

While debugging this issue, I added some trace statements and confirmed that both [OneTimeSetUp] methods ran and completed, but no [Test] methods were called.

The simplified code below reproduces it pretty reliably (but not every time!) If you comment out the Thread.Sleep(), the problem goes away. The problem also goes away if you turn off parallelization, or if parallelization is enabled with only a single [SetUpFixture].

using System;
using System.Threading;
using NUnit.Framework;

[assembly: Timeout(60000)]
[assembly: Parallelizable(ParallelScope.Children)]

namespace ParallelSetUpFixtures
{
    [SetUpFixture]
    public class FooSetUp
    {
        [OneTimeSetUp]
        public void FooSetUpMethod()
        {
            Console.WriteLine("FooSetUp");
        }
    }

    // [Parallelizable(ParallelScope.None)]  // if this line is uncommented, then NUnit will not hang
    [SetUpFixture]
    public class BarSetUp
    {
        [OneTimeSetUp]
        public void BarSetUpMethod()
        {
            Thread.Sleep(500); // if this line is commented out, then NUnit will not hang
            Console.WriteLine("BarSetUp");
        }
    }

    [TestFixture]
    public class MyTests
    {
        [Test]
        public void MyTest()
        {
            Console.WriteLine("MyTest");
        }
    }
}

Here is the command line I'm using:

nunit3-console.exe parallel_set_up_fixtures-w64r-16-3.dll --framework=4.5

While it is hung, I see three threads with NUnit code in the call stacks. The other threads are just worker threads waiting for work. Here are partial stack traces for the three NUnit threads:

    mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne    Unknown
    mscorlib.dll!System.Threading.WaitHandle.WaitOne    Unknown
>   nunit.engine.dll!NUnit.Engine.Agents.RemoteTestAgent.WaitForStop Line 109   C#
    nunit-agent.exe!NUnit.Agent.NUnitTestAgent.Main Line 133    C#

    mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne    Unknown
    mscorlib.dll!System.Threading.WaitHandle.WaitOne    Unknown
>   nunit.framework.dll!NUnit.Framework.Api.NUnitTestAssemblyRunner.WaitForCompletion Line 236  C#
    nunit.framework.dll!NUnit.Framework.Api.NUnitTestAssemblyRunner.Run Line 200    C#
    nunit.framework.dll!NUnit.Framework.Api.FrameworkController.RunTests Line 208   C#
    nunit.framework.dll!NUnit.Framework.Api.FrameworkController.RunTestsAction.RunTestsAction Line 404  C#

    // the size of queue is 0
>   nunit.framework.dll!NUnit.Framework.Internal.Execution.EventQueue.Dequeue Line 274  C#
    nunit.framework.dll!NUnit.Framework.Internal.Execution.EventPump.PumpThreadProc Line 208    C#
    mscorlib.dll!System.Threading.ExecutionContext.RunInternal  Unknown

[CharliePoole]

Looks like a good test case for resolving this. Even though we allow multiple SetUpFixtures in the same namespace, I didn't give much thought to how this would work in a parallel situation. For now, I suggest you combine your code into a single SetUpFixture.

[CharliePoole]

Note: #1208 is another bug that will make the situation even worse. Do not use use the STA for your SetUpFixtures.

[CharliePoole]

@RobinsonWM Not sure if this is still an issue for you, but I'd like to clean it up.

Can you explain if there is anything special you are trying to achieve in having two SetUpFixtures in the same namespace with one parallelizable and the other non-parallelizable? Essentially, they are equivalent to a combined SetUpFixture with the two parts running in an indeterminate order. So it's not entirely clear what would be the "right" thing for us to do in this case.

In my current working code, I have verified that they can run correctly when both are parallel or both are non-parallel. Your hang is probably due to the conflicting attributes, but I haven't proven that yet.

BTW, this is being worked on in PR #2476

[RobinsonWM]

I worked around it by combining them into a single SetUpFixture and calling Task.Run() multiple times in the OneTimeSetUp method.

My intention was for both SetUpFixtures to be parallelizable. I only included the [Parallelizable(ParallelScope.None)] in my code sample to illustrate that, if you uncommented that line, it would work around the problem. But I didn't intend to do that in my project.

I originally wrote the two SetUpFixtures to be completely independent (one was related to initializing logging, and other was related initializing some unmanaged resources that my test used.) Once I realized that wouldn't work, combining them into one class was an acceptable workaround; it just made for a larger code file with unrelated concerns in it.

[CharliePoole]

NUnit's structure of tests is a tree, with each node having only one parent. If we were to allow the two SetUpFixtures to run in parallel with one another, they would both need to be parents of the contained TestFixtures, transforming the tree into a more general directed network. So what we do is make one of the SetUpFixtures the parent of the other one, which in turn is the parent of the TestFixtures. The Parallelizable attribute, therefore, only applies to other setup fixtures and test fixtures. In my relatively simple tests, having the two setup fixtures sue different parallel scopes works, but of course requires a lot more switching between work queues to get the job done. So making only one of them parallelizable actually slows things down a bit.

What, if anything do you think we should do about this? One approach would be to force the two setup fixtures to use the same settings and give an error or warning if they are in conflict. That may be a bit heavy-handed, however. @nunit/framework-team What do you think?

BTW, none of the above issues apply to setup fixtures at two different levels of the namespace hierarchy.

[CharliePoole]

@nunit/framework-team I'm indicating in #2476 that it fixes this issue. We should review to see if more work is needed or a separate issue should be created.

In the fix, marking the two setup fixtures as parallelizable doesn't hang NUnit any longer. However, it doesn't have the effect that @RobinsonWM actually wanted: to allow the two SetUpFixtures to run in parallel. Currently, that is not possible for two setup fixtures in the same namespace.

[ChrisMaddock]

Re: different levels of parallelisation - there’s nothing actually unexpected here, is there? Nunit is essentially running one in parallel and one in non-parallel, that’s just an odd thing to be doing? I don’t see the need for an error message there.

Regarding multiple parallel setup fixtures not running in parallel - I think that one is more surprising, but I’m not sure of an easy was either to do that, or to flag a warning for it...

[CharliePoole]

@ChrisMaddock Essentially, I think the problem is that most users will assume that two SetUpFixture in the same namespace, marked as Parallelizable, will run in parallel. We should add to the docs to clarify that.

Source of the problem is that we model this internally as one fixture containing the other, which is not what the user sees. Changing the internal design is possible but also non-trivial and it's not clear that the usage is common.

[ChrisMaddock]

I agree with you on both points. My previous comment on 'nothing unexpected' was referring to your first question, about when a user has one SetUpFixture parallelizable, and one non-parallelizable. 😄