NullReferenceException after parallel tests have finished executing

[sanoursa]

Using NUnit 3.9.0, we have enabled Parallelizable(ParallelScope.Fixtures), and while the tests all run correctly, about 30% of the time the test suite fails because NUnit's background thread throws a NullReferenceException with the following stack:

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object. at 
NUnit.Framework.Internal.Execution.WorkItemQueue.get_IsEmpty() at 
NUnit.Framework.Internal.Execution.WorkShift.get_HasWork() at 
NUnit.Framework.Internal.Execution.ParallelWorkItemDispatcher.SelectNextShift() at 
NUnit.Framework.Internal.Execution.ParallelWorkItemDispatcher.OnEndOfShift(WorkShift endingShift) at 
NUnit.Framework.Internal.Execution.WorkShift.<StartWorkers>b__26_1(TestWorker s, WorkItem ea) at 
NUnit.Framework.Internal.Execution.TestWorker.TestWorkerThreadProc() at 
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at 
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at 
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at 
System.Threading.ThreadHelper.ThreadStart() 

Is this a known issue? Is there anything I can do to gather more information to help diagnose the problem?

[CharliePoole]

I have seen this on occasion as well. I don't think it's separately documented.

@sanoursa You may want to try the latest framework build from our development feed at https://www.myget.org/feed/nunit/package/nuget/NUnit

[jnm2]

Huh, thought that stack trace was one of the two fixed in 3.9 by #2428.

[sanoursa]

@CharliePoole I tried the latest package from the dev feed, and still hit the same exception. Is there any extra logging that I can enable to help diagnose this?

Also I've noticed that I hit the exception very reliably on a machine with 4 cores, and very infrequently on a machine with 2 cores.

[CharliePoole]

It's tricky because we end up with a list of queues that includes null values. That causes an exception much later but I can't see how it happens.

[sanoursa]

I'm still trying to understand the code, but is it possible that there's a race between WorkItemQueue.Save() calling InitializeQueues(), and the background thread calling IsEmpty?

While InitializeQueues is in progress, _innerQueues is in a state where some of its items can be null. If anything tries to access _innerQueues during that time, it could definitely hit this issue.

What if InitializeQueues is updated to create a new ConcurrentQueue in a local variable, and only assign it to _innerQueues once it's been fully initialized?

[sanoursa]

I'm testing out that theory now and I'll report back on the results. I have a very reliable repro, so if you have any other theories you'd like to test out, I'd be happy to try those too.

[sanoursa]

So far, so good. I've done several test runs on both 2 CPU and 4 CPU machines, with no hits on this exception so far. I've scheduled 10 more runs, so I'll check on the results in the morning and, if all goes well, I'll send you a PR with my fix.

[sanoursa]

Thanks for taking my PR! How soon do you think we will get this fix in an official release?

[jnm2]

Here's the milestone: https://github.com/nunit/nunit/milestone/34

If you need an immediate workaround in a project, you can merge this with your project's nuget.config and upgrade NUnit to version 3.10.0-dev-05190 or later:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="NUnit MyGet" value="https://www.myget.org/F/nunit/api/v3/index.json" />
  </packageSources>
</configuration>

@rprouse Are you still thinking Feb 28?

[jnm2]

GitHub's not smart enough to recognize Fixes issue # like it does with Fixes #. 😄 Closed in d355ad2 via #2668