Parallel execution of test scenarios within a feature with NUnit 3

[bedegaming-aleksej]

Background

I am working on a test solution that utilizes a combination of NUnit v3.7.1 and SpecFlow v2.2.0.

NUnit 3 now supports running test cases within a fixture in parallel with each other by setting ParallelScope to Children on an assembly level.

Tests are run in a single AppDomain.

Problem
This seems to kill scenario context injection with following errors:

TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at BoDi.ObjectContainer.Resolve(Type typeToResolve, ResolutionList resolutionPath, String name) in C:\projects\specflow\TechTalk.SpecFlow\BoDi\BoDi.cs:line 697

System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
  ----> System.Collections.Generic.KeyNotFoundException : The given key was not present in the dictionary.
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BoDi.ObjectContainer.CreateObject(Type type, ResolutionList resolutionPath, RegistrationKey keyToResolve) in C:\projects\specflow\TechTalk.SpecFlow\BoDi\BoDi.cs:line 800
   at BoDi.ObjectContainer.TypeRegistration.Resolve(ObjectContainer container, RegistrationKey keyToResolve, ResolutionList resolutionPath) in C:\projects\specflow\TechTalk.SpecFlow\BoDi\BoDi.cs:line 341

Issue appears to occur when constructing ScenarioContext.
Note
Tests from different features are running in parallel with each other just fine when ParallelScope is set to Fixtures.

[bedegaming-aleksej]

#657 could be related.

[SabotageAndi]

Could it be related to #884 ?

Do you have a project where we can reproduce this issue?

[abasau]

@SabotageAndi I faced the same issue today. Here is a sample project that can be used to reproduce the issue: NUnitExample.zip
Just run runtests.cmd.
It doesn't seem to be related to #884 as the issues have different stack traces and this one doesn't reproduce when [assembly: LevelOfParallelism(1)].

[abasau]

@SabotageAndi Indeed. The issue is the same as #884. Although the original description of #884 is not correct because the issue will not reproduce with single thread and will reproduce even if the number of threads more than number of tests.
Feel free to close this issue if there are no objections from others and we will continue under #884.
I left a comment about the nature of the issue there.

[unickq]

Faced with the same issue:
NUnit:ParallelScope.Children - can't run 2 scenarios at the same time with the errors described above

[SabotageAndi]

It's still happening with the latest version of NUnit and SpecFlow.
@gasparnagy could you have a look at it?

[gasparnagy]

@SabotageAndi not yet, unfortunately.

[bambos-xiouros]

Just to add, it's still happening with SpecFlow 2.3.2 and NUnit 3.10.1.

Cheers.

[brian-menzies]

Just to add, it's still happening with SpecFlow 2.4.1 and NUnit 3.11.0.

[ClassyCircuit]

Any plans to fix it?

[SabotageAndi]

I don't have it on my work list.