Random test result using nunit + parallel execution #884

elias551 · 2017-07-04T15:59:32Z

Hello,

I have reproduced an error that occured randomly on my CI server:
https://github.com/elias551/specflow-nunit-parallelism-error-poc

I'm using specflow + nunit. It seems like the tests executed in parallel tend to fail often.

In the example I have 10 tests, that I allow to execute in parallel (no tests are dependent to each other), if my machine have 8 cores, then the first 8 tests executed will have the correct result. But then, when a core is freed, the test that will be executed next may throw with the following error:

System.NullReferenceException : La référence d'objet n'est pas définie à une instance d'un objet.
   à TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.GetStepMatch(StepInstance stepInstance)
   à TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance)
   à TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()
   à TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()
   à ClassLibrary1.SpecFlowFeature8Feature.ScenarioCleanup()
   à ClassLibrary1.SpecFlowFeature8Feature.AddTwoNumbers() dans C:\Users\eplatek\Documents\Visual Studio 2017\Projects\ClassLibrary1\ClassLibrary1\SpecFlowFeature8.feature:ligne 11

This error will happen more often if you reduce the level of parallelism with this:
[assembly: LevelOfParallelism(1)]

On the contrary, if you increase the level of parallelism to a greater value than the total number of tests, the tests will always have the correct value.

Given my test project I was not able to define a greater number than the total number of tests (> 50), because in this case every test run at the same time, and nunit never returns a result.

The text was updated successfully, but these errors were encountered:

Peter-B- · 2017-07-06T10:56:30Z

Hello,

I see the same problem.

When I set [assembly: LevelOfParallelism(1)], all unit tests pass.

When I set [assembly: LevelOfParallelism(4)], a lot of tests fail. But never the unit tests that run with the first batch.

When I set [assembly: LevelOfParallelism(400)], all unit tests are started at the same time and all pass.

I use [assembly: Parallelizable(ParallelScope.Fixtures)].

I did not see this behaviour with the last preview nuget packet of 2.2.

Does anybody have an idea where this could come from?

SabotageAndi · 2017-07-06T11:06:25Z

@gasparnagy Do you have an idea?

gasparnagy · 2017-07-11T08:43:45Z

@SabotageAndi No...

@elias551 @Peter-B- Could you please try to place the symbol file (.pdb) next to the techtalk.specflow.dll in order to see a line number for the failure? The sybbol files for v2.2 can be downloaded from here.

elias551 · 2017-07-11T09:18:48Z

@gasparnagy the dll I got from nuget do not match your pdbs, I have the same stack trace.

I retried with the last source in the repo, I still have null reference exception but not at the same place:

System.NullReferenceException : La référence d'objet n'est pas définie à une instance d'un objet.
   à TechTalk.SpecFlow.Infrastructure.ContextManager.InternalContextManager`1.DisposeInstance() dans C:\Users\eplatek\Downloads\SpecFlow-master\TechTalk.SpecFlow\Infrastructure\ContextManager.cs:ligne 52
   à TechTalk.SpecFlow.Infrastructure.ContextManager.InternalContextManager`1.Init(TContext newInstance, IObjectContainer newObjectContainer) dans C:\Users\eplatek\Downloads\SpecFlow-master\TechTalk.SpecFlow\Infrastructure\ContextManager.cs:ligne 34
   à TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeFeatureContext(FeatureInfo featureInfo) dans C:\Users\eplatek\Downloads\SpecFlow-master\TechTalk.SpecFlow\Infrastructure\ContextManager.cs:ligne 172
   à TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnFeatureStart(FeatureInfo featureInfo) dans C:\Users\eplatek\Downloads\SpecFlow-master\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:ligne 95
   à TechTalk.SpecFlow.TestRunner.OnFeatureStart(FeatureInfo featureInfo) dans C:\Users\eplatek\Downloads\SpecFlow-master\TechTalk.SpecFlow\TestRunner.cs:ligne 41
   à ClassLibrary1.SpecFlowFeature9Feature.FeatureSetup()

The behavior is quite the same, the first batch runs fine. Then the start of some of the next fail.

It seems like the method ContextManager.Init disposes the current instance if it is not null. But another thread may dispose it before on a race condition. Adding a lock solves the problem in my case:


public void Init(TContext newInstance, IObjectContainer newObjectContainer)
{
    lock (LockValue)
    {
        if (instance != null)
        {
            testTracer.TraceWarning(string.Format("The previous {0} was not disposed.", typeof(TContext).Name));
            DisposeInstance();
        }
        instance = newInstance;
        objectContainer = newObjectContainer;
    }
}

Even if it solves the same problem, the root cause is not the same because the stack traces are very different, so I can't say if it is a good solution.

gasparnagy · 2017-07-11T09:32:41Z

@elias551 thx for the info. And if you put the lock into the context manager, do you still get the other error or with this "fix" the other error also goes away?

gasparnagy · 2017-07-11T09:39:01Z

@elias551 one more thing: how does the test output look like (where is shows the executed steps and the called step definition methods)? Do you see things like #5: Given ... or just Given ...?

Peter-B- · 2017-07-11T09:50:45Z

Hello

I copied the pdbs and I get NullReferenceExceptions with differen stacks:

System.NullReferenceException : Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
   bei TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.GetStepMatch(StepInstance stepInstance) in C:\projects\specflow\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:Zeile 364.
   bei TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance) in C:\projects\specflow\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:Zeile 295.
   bei TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep() in C:\projects\specflow\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:Zeile 131.
   bei Qm.Pcb.Logic.Tests.DbServiceTests.UserDbServiceTests.CreateUserFeature.ScenarioCleanup()
   bei Qm.Pcb.Logic.Tests.DbServiceTests.UserDbServiceTests.CreateUserFeature.CreateNewPcbUserWithRolesRolesAreReturned() in D:\Projekte\Osram QM\PlatinenDB\Tests\Qm.Pcb.Logic.Tests\DbServiceTests\UserDbServiceTests\CreateUser.feature:Zeile 46.

and

System.NullReferenceException : Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
   bei TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo) in C:\projects\specflow\TechTalk.SpecFlow\Infrastructure\ContextManager.cs:Zeile 183.
   bei TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioStart(ScenarioInfo scenarioInfo) in C:\projects\specflow\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:Zeile 125.
   bei Qm.Pcb.Logic.Tests.DbServiceTests.UserDbServiceTests.CreateUserFeature.ScenarioSetup(ScenarioInfo scenarioInfo)
   bei Qm.Pcb.Logic.Tests.DbServiceTests.UserDbServiceTests.CreateUserFeature.CreateNewPcbUserWithRolesRolesAreStored() in D:\Projekte\Osram QM\PlatinenDB\Tests\Qm.Pcb.Logic.Tests\DbServiceTests\UserDbServiceTests\CreateUser.feature:Zeile 64.

My steps all look like:

Scenario: Create new PcbUser with roles: Roles are stored
	Given I have these MdmUsers
	| WinLogin      | FirstName | LastName | LocationId |
	| darkwing.duck | Darkwing  | Duck     | Regensburg |
	When I call UserDbService.Create as "admin" with
	| WinLogin      | LocationId | UserRoles           |
	| darkwing.duck | Regensburg | ReadData, EditUsers |
	Then I expect these PcbUserRoleAssignments in the DB
	| Role      | WinLogin      |
	| ReadData  | darkwing.duck |
	| EditUsers | darkwing.duck |

I have no #5.

Does that help you?

Peter-B- · 2017-07-11T09:52:42Z

@gasparnagy
The first stack trace seems to come up with the first failing test. The second comes for all following scenarios.

elias551 · 2017-07-11T09:53:00Z

@gasparnagy @Peter-B- my bad, my project was still using 2.1.0, so I updraded to 2.2.0, and the stack traces are now the same as Peter's.

gasparnagy · 2017-07-11T10:07:58Z

@Peter-B- @elias551 thx. could you make a comment on my question related to the test output? #884 (comment)

Peter-B- · 2017-07-11T10:17:59Z

@gasparnagy Sorry, I didn't read properly. When i switch to parallel execution, I don't see the outputs any more. Sometimes I see my own Console.WriteLine outputs, but I can't find any information from SpecFlow.

gasparnagy · 2017-07-11T11:28:24Z

:( my suspicion is that SpecFlow does not recognize properly that NUnit runs the tests in parallel and it does not initialize the necessary thread safe infrastructures. (e.g. it creates a ContextManager per thread, so the tests will not access it concurrently).

Normally, parallel execution is detected if the test runner initialized by the test (at TechTalk.SpecFlow.TestRunnerManager.GetTestRunner() in the generated code behind file) is initialized from multiple threads. Maybe NUnit initializes the tests from the same thread (?) and only spawns the execution to different threads... Sounds strange though...

I will think how to test this...

elias551 · 2017-07-11T11:31:27Z

Indeed logging is strange, only the first running test has the output and it contains the traces of every test:

#21: Given I have entered 50 into the calculator
#19: Given I have entered 50 into the calculator
#24: Given I have entered 50 into the calculator
#20: Given I have entered 50 into the calculator
#25: Given I have entered 50 into the calculator
#23: Given I have entered 50 into the calculator
#18: Given I have entered 50 into the calculator
#22: Given I have entered 50 into the calculator
-> #21: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
-> #23: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
-> #18: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
-> #22: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
-> #19: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
-> #25: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
-> #24: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
-> #20: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(50) (1,0s)
#24: And I have entered 70 into the calculator
#19: And I have entered 70 into the calculator
#23: And I have entered 70 into the calculator
#20: And I have entered 70 into the calculator
#21: And I have entered 70 into the calculator
#22: And I have entered 70 into the calculator
#18: And I have entered 70 into the calculator
#25: And I have entered 70 into the calculator
-> #25: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
-> #22: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
-> #23: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
-> #20: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
-> #18: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
-> #24: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
-> #21: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
-> #19: done: SpecFlowFeature1Steps.GivenIHaveEnteredIntoTheCalculator(70) (1,0s)
#18: When I press add
#22: When I press add
#19: When I press add
#23: When I press add
#21: When I press add
#20: When I press add
#25: When I press add
#24: When I press add
-> #23: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
-> #21: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
-> #20: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
-> #22: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
#23: Then the result should be 120 on the screen
-> #19: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
#21: Then the result should be 120 on the screen
#20: Then the result should be 120 on the screen
#19: Then the result should be 120 on the screen
#22: Then the result should be 120 on the screen
-> #24: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
#24: Then the result should be 120 on the screen
-> #18: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
#18: Then the result should be 120 on the screen
-> #23: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #22: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #18: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #20: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #21: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #25: done: SpecFlowFeature1Steps.WhenIPressAdd() (0,0s)
#25: Then the result should be 120 on the screen
-> #25: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #24: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #19: done: SpecFlowFeature1Steps.ThenTheResultShouldBeOnTheScreen(120) (0,0s)
-> #18: warning: The previous FeatureContext was not disposed.

The first failing test has the following line in logging:
-> Using default config

Peter-B- · 2017-07-11T11:33:09Z

@gasparnagy Thanks a lot for your effort.

I hope I find some time to take a deeper into this issue later this week. Maybe I can dig out something that might help you.

gasparnagy · 2017-07-11T12:55:11Z

@Peter-B- I will try to figure it out. The logs from @elias551 basically proved that SpecFlow does operate in parallel mode (because you see the thread numbers in the log), so the issue must be something different. Also the last warning on thread 18 is very strange... I will check it.

gasparnagy · 2017-07-28T13:59:03Z

@elias551 @Peter-B- There are some hints in the related issues (#869, #902, #913) that would be worth trying.

check if the issue is related to the NUnit version. FeatureContext throws a NullReferenceException #869 mentions that it works on NUnit 3.6.1, but not with nunit 3.7.1
check if the issue is solved when you put the [NUnit.Framework.SingleThreaded] attribute to the generated feature class manually. [NUnit generator] Add SingleThreaded Attribute to feature class #913 has more info on this.

Could you please check these?

elias551 · 2017-07-28T15:54:09Z

I ran the tests a few times with the attribute, indeed there is no more errors! I still have all of the logs in the first test though

elias551 · 2017-07-28T15:55:38Z

My nunit version is 3.7.1 btw

gasparnagy · 2017-07-28T18:03:28Z

@elias551 and are they still run parallel? Because the documentation of that attribute does not say much about what it is good for actually.

Peter-B- · 2017-07-28T20:06:52Z

@gasparnagy Downgrading to NUnit 3.6.1 indeed solved the problem.

I injected [NUnit.Framework.SingleThreaded] in all my *.feature.cs files, but that didn't change the change the behavior: The tests still run in parallel and the NullReferenceExceptions keep coming. (That is when using NUnit 3.7.1)

Peter-B- · 2017-08-31T13:27:21Z

I updated to NUnit 3.8.1 today. That fixed the issue for me.

@elias551 If updating NUnit solves the problem for you also, I think we can close the issue.

abasau · 2017-11-04T01:06:41Z

@gasparnagy I have checked your assumption from #884 (comment) and it's correct. When running tests from the same fixture in parallel, NUnit creates one instance of the fixture class, calls FeatureSetup (method with TestFixtureSetUpAttribute) and then runs the tests in parallel. All tests (even those that are running in different threads) get the same TestRunner.

There is also an issue on SpecFlow side - testRunner is storied in static field for NUnit:

private static TechTalk.SpecFlow.ITestRunner testRunner;

I have also checked NUnit 3.8.1 and the issue is still reproducible.
TestFixtureSetUpAttribute and TestFixtureTearDownAttribute were removed in 3.8.1 so the following workaround might be required:

namespace NUnit.Framework
{
    public class TestFixtureSetUpAttribute : OneTimeSetUpAttribute
    {
    }

    public class TestFixtureTearDownAttribute : OneTimeTearDownAttribute
    {
    }
}

RajaAutomation · 2019-03-06T15:04:58Z

@aliaksandrbasau @gasparnagy can you please let me know if this issue is still open or has been resolved as part of any specflow release.

RajaAutomation · 2019-03-06T15:06:00Z

@gasparnagy I have checked your assumption from #884 (comment) and it's correct. When running tests from the same fixture in parallel, NUnit creates one instance of the fixture class, calls FeatureSetup (method with TestFixtureSetUpAttribute) and then runs the tests in parallel. All tests (even those that are running in different threads) get the same TestRunner.

There is also an issue on SpecFlow side - testRunner is storied in static field for NUnit:
private static TechTalk.SpecFlow.ITestRunner testRunner;
I have also checked NUnit 3.8.1 and the issue is still reproducible.
TestFixtureSetUpAttribute and TestFixtureTearDownAttribute were removed in 3.8.1 so the following workaround might be required:
namespace NUnit.Framework
{
    public class TestFixtureSetUpAttribute : OneTimeSetUpAttribute
    {
    }

    public class TestFixtureTearDownAttribute : OneTimeTearDownAttribute
    {
    }
}

@aliaksandrbasau can you please let me know if the work around you have provided is available in any release version of NUnit.

RajaAutomation · 2019-03-06T16:07:33Z

@gasparnagy will we also face same issue if we use SPECRUN as an unit test provider instead of NUNIT.

abasau · 2019-03-07T01:52:49Z

@RajaAutomation I have checked the latest stable release of SpecFlow (2.4.1) and both issues I mentioned earlier (testRunner field being static and wrong fixture setup/teardown attributes) are resolved.
Although I didn't check whether the original issue is reproducible.
SPECRUN didn't have the issue because it's NUnit-specific and occurs only when NUnit runs tests (not fixtures) in parallel.

RajaAutomation · 2019-03-07T09:19:26Z

@RajaAutomation I have checked the latest stable release of SpecFlow (2.4.1) and both issues I mentioned earlier (testRunner field being static and wrong fixture setup/teardown attributes) are resolved.
Although I didn't check whether the original issue is reproducible.
SPECRUN didn't have the issue because it's NUnit-specific and occurs only when NUnit runs tests (not fixtures) in parallel.

@aliaksandrbasau Thank you for the quick response. Currently we are using specflow 2.3.2 and the issue is occurring.
Any idea if this is the issue with my hooks setup or is it the expected behavior of specflow 2.3.2

abasau · 2019-03-07T14:17:29Z

@RajaAutomation If you are talking about the original issue (random test failures when tests are being executed in parallel), than it's not an issue with your hooks setup nor expected behavior of specflow but a defect in specflow.
But I would double-check that this issue really describes the issue you encountered. If adding [assembly: Parallelizable(ParallelScope.Fixtures)] to AssemblyInfo.cs resolves the issue than this issue indeed is affecting your tests.

RajaAutomation · 2019-03-07T14:24:35Z

@RajaAutomation If you are talking about the original issue (random test failures when tests are being executed in parallel), than it's not an issue with your hooks setup nor expected behavior of specflow but a defect in specflow.
But I would double-check that this issue really describes the issue you encountered. If adding [assembly: Parallelizable(ParallelScope.Fixtures)] to AssemblyInfo.cs resolves the issue than this issue indeed is affecting your tests.

@aliaksandrbasau ....
Here is the problem i am facing

I have multiple feature files and each feature file has multiple scenario outlines and each scenario outline has multiple examples.

When i use Parallelizable(ParallelScope.Fixtures)], one example of scenario outline 1 of feature A runs in parallel with another example of scenario outline 1 of feature B.
However my intention is to run the features in the following way:--

Feature A --> Scenario Outline 1 -->Run all examples in parallel, then
Feature A -- Scenario Outline 2 ->Run all examples in parallel, then
Feature B --> Scenario Outline 1 --> Run all examples in parallel
Feature B --> Scenario Outline 2 --> Run all examples in parallel and so on....

abasau · 2019-03-07T15:00:30Z

@RajaAutomation And if you are running Tests in parallel instead of Fixtures, you get random test failures, right?

RajaAutomation · 2019-03-07T15:15:52Z

@RajaAutomation And if you are running Tests in parallel instead of Fixtures, you get random test failures, right?

All tests fail with one of the below reasons -

System.NullReferenceException : Object reference not set to an instance of an object.
System.ArgumentException : An item with the same key has already been added.

abasau · 2019-03-07T15:21:03Z

@RajaAutomation Yep. You've found the right issue. Unfortunately there is no fix or workaround for it besides running Fixtures in parallel instead of Tests.

elias551 · 2019-03-07T15:29:19Z

Hello all, I'm not working anymore on the project where I was facing this problem, but I can confirm that adding the Parallelizable attribute in AssemblyInfo.cs solved my issue

RajaAutomation · 2019-03-07T16:23:17Z

@RajaAutomation Yep. You've found the right issue. Unfortunately there is no fix or workaround for it besides running Fixtures in parallel instead of Tests.

@aliaksandrbasau Thank you for the information.
Do you if anyone from specflow team is looking into this issue. @gasparnagy @aliaksandrbasau

abasau · 2019-03-07T17:48:55Z

@RajaAutomation I'm not a part of specflow team so I don't really know.

gasparnagy · 2019-03-08T13:22:55Z

@RajaAutomation after scanning through the discussion, I think what you should check first if the issue still exsits if you upgrade to SpecFlow v2.4.1, because there is a good chance that is has been fixed.

RajaAutomation · 2019-03-11T10:52:06Z

@gasparnagy I have upgraded the specflow to v2.4.1 and for some reason i am not able to see the tests in the UNIT TEST EXPLORER.
I am using the NUnit as the unittestprovider.

Is this a known issue.

gasparnagy · 2019-03-11T11:25:18Z

@RajaAutomation No. It should work. Make sure you have the latest NUnit visual studio test adapter. (Restart of VS might be also needed).

RajaAutomation · 2019-03-11T13:22:36Z

@gasparnagy I have checked it and i am using NUnit Visual Studio Test Adapter (3.13.0) which is the latest stable version available.
Restarting of Visual Studio did not give me any results.

Stephen-Mc · 2019-03-11T13:55:22Z

According to this issue on Stack Overflow, this can happen if there is a mismatch with the bit version of your target platform and the architecture specified under Test -> Test Settings -> Default Processor Architecture. There's quite a lot of posts there (some more recent than others), so might be worth reading through to see if there is any information of use to your case.

RajaAutomation · 2019-03-11T15:39:53Z

@Stephen-Mc Implemented all the posts available in that forum however issue still persists.

@gasparnagy @Stephen-Mc I have found similar open issue reported when specflow upgraded to v2.3.2 from v2.2.1 (#1201) . In my case its an upgrade from v2.3.2 to v2.4.1.

RajaAutomation · 2019-03-11T16:06:34Z

@gasparnagy @Stephen-Mc After adding SpecFlowSingleFileGenerator as the custom tool value in feature file properties the tests are visible in test explorer.

RajaAutomation · 2019-03-11T16:06:58Z

@RajaAutomation after scanning through the discussion, I think what you should check first if the issue still exsits if you upgrade to SpecFlow v2.4.1, because there is a good chance that is has been fixed.

@gasparnagy Issue still persists

MattKeenum · 2019-03-26T20:10:26Z

NUnit3TestAdapter 3.12.0
NUnit 3.11.0
SpecFlow 2.4.1

If, in your assembly, you have [assembly: Parallelizable(ParallelScope.Fixtures)] then it will correctly run test fixtures in parallel.

If, in your assembly, you have [assembly: Parallelizable(ParallelScope.Children)] then it will give similar error messages to those referenced above if the tests selected to run are in the same fixture.
If the tests selected to run in parallel are in different fixtures, everything works.

Will we get the ability to run parallelization at the test level, rather than the fixture level?

SabotageAndi · 2019-03-27T07:41:25Z

@MattKeenum Because how SpecFlow work, parallelization on test level will not work without a lot of rewrite.

MattKeenum · 2019-03-27T10:51:49Z

@SabotageAndi Thank you for clarifying. Is SpecFlow being rewritten for .Net Core? Or can we expect the same behavior there as well?

SabotageAndi · 2019-03-27T11:16:11Z

There was no rewrite for .NET Core. It will be the same behaviour in SpecFlow 3 as in SpecFlow 2.

ClassyCircuit · 2019-04-11T05:48:31Z

@SabotageAndi Are there any workarounds to get this working with NUnit? I have a lot of scenario outlines that should run in parallel, but I get the same errors as others mentioned.

If it's not possible at all with NUnit, is there another runner besides Specrun that can handle this?

I read that XUnit doesn't run datarow tests in parallel at all. How about mstest?

SabotageAndi · 2019-04-11T10:39:26Z

@janissimsons No there isn't a workaround. Other test runners have also the same problem. The issue is with SpecFlow.

MattKeenum · 2019-04-11T10:59:53Z

@janissimsons We are using NUnit and we are writing a custom SingleFileGenerator to generate a .cs file like the SpecFlow does, but still utilizing SpecFlow's extension for the Step Definition generation and feature file because SpecFlow does an amazing job at that.

We'll also have to write a back end tool to connect the auto-generated Step Definitions with our auto-generated NUnit tests, but that shouldn't be difficult. It won't have all of the features that SpecFlow has, but we are hoping we can run methods in parallel from this. It's purely exploratory right now.

Prateek1908 · 2020-11-04T10:19:19Z

Any update . I am using .net core along with Specflow and getting object reference error

SabotageAndi · 2020-11-04T12:35:46Z

@Prateek1908 No

SabotageAndi added NUnit Runtime labels Jul 6, 2017

This was referenced Jul 14, 2017

Scenarios fail when MsTest is set to use parallel execution #448

Closed

Parallel execution of test scenarios within a feature with NUnit 3 #894

Open

gasparnagy mentioned this issue Jul 28, 2017

[NUnit generator] Add SingleThreaded Attribute to feature class #913

Open

SabotageAndi added Difficulty: hard Severity: medium Bug labels Nov 2, 2020

Random test result using nunit + parallel execution #884

Random test result using nunit + parallel execution #884

Comments

elias551 commented Jul 4, 2017

Peter-B- commented Jul 6, 2017

SabotageAndi commented Jul 6, 2017

gasparnagy commented Jul 11, 2017

elias551 commented Jul 11, 2017 • edited

gasparnagy commented Jul 11, 2017

gasparnagy commented Jul 11, 2017

Peter-B- commented Jul 11, 2017

Peter-B- commented Jul 11, 2017

elias551 commented Jul 11, 2017

gasparnagy commented Jul 11, 2017

Peter-B- commented Jul 11, 2017

gasparnagy commented Jul 11, 2017

elias551 commented Jul 11, 2017

Peter-B- commented Jul 11, 2017

gasparnagy commented Jul 11, 2017

gasparnagy commented Jul 28, 2017

elias551 commented Jul 28, 2017

elias551 commented Jul 28, 2017

gasparnagy commented Jul 28, 2017

Peter-B- commented Jul 28, 2017 • edited

Peter-B- commented Aug 31, 2017

abasau commented Nov 4, 2017

RajaAutomation commented Mar 6, 2019

RajaAutomation commented Mar 6, 2019

RajaAutomation commented Mar 6, 2019

abasau commented Mar 7, 2019

RajaAutomation commented Mar 7, 2019

abasau commented Mar 7, 2019

RajaAutomation commented Mar 7, 2019

abasau commented Mar 7, 2019

RajaAutomation commented Mar 7, 2019

abasau commented Mar 7, 2019

elias551 commented Mar 7, 2019

RajaAutomation commented Mar 7, 2019

abasau commented Mar 7, 2019

gasparnagy commented Mar 8, 2019

RajaAutomation commented Mar 11, 2019

gasparnagy commented Mar 11, 2019

RajaAutomation commented Mar 11, 2019

Stephen-Mc commented Mar 11, 2019

RajaAutomation commented Mar 11, 2019

RajaAutomation commented Mar 11, 2019 • edited

RajaAutomation commented Mar 11, 2019 • edited

MattKeenum commented Mar 26, 2019 • edited

SabotageAndi commented Mar 27, 2019

MattKeenum commented Mar 27, 2019

SabotageAndi commented Mar 27, 2019

ClassyCircuit commented Apr 11, 2019

SabotageAndi commented Apr 11, 2019

MattKeenum commented Apr 11, 2019

Prateek1908 commented Nov 4, 2020

SabotageAndi commented Nov 4, 2020

elias551 commented Jul 11, 2017 •

edited

Peter-B- commented Jul 28, 2017 •

edited

RajaAutomation commented Mar 11, 2019 •

edited

RajaAutomation commented Mar 11, 2019 •

edited

MattKeenum commented Mar 26, 2019 •

edited