New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specflow + Nunit 3.7.1 = Exception #919

Closed
shadowfoxish opened this Issue Aug 9, 2017 · 8 comments

Comments

Projects
None yet
5 participants
@shadowfoxish

shadowfoxish commented Aug 9, 2017

Hi team, we have a situation where if we update NUnit from 3.6.1 to 3.7.1, our Specflow tests break. I really don't know if this is a Specflow issue or an NUnit issue, but my gut says its Specflow related.

I created a VS2017 project to demonstrate the issue:
https://github.com/shadowfoxish/Temp-NunitIssue
Build the project as it is, and try to run the tests from Visual Studio Test Explorer. They'll both fail and give this error
Object reference not set to an instance of an object.System.NullReferenceException: Object reference not set to an instance of an object. at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, String message, Object[] args) (full stack trace on the readme.md file in the above repo).

If you go to Nuget package explorer and change the referenced version of Nunit.Framework to 3.6.1, then the tests succeed. (You may have to remove the binding redirect in app.config if you downgrade the version number).

We're not on the latest and greatest version of Specflow (shame on us) but our automation guy is only currently licensed for that previous version (2.1.0). Thoughts?

We found this in research: nunit/nunit#2267

@Stephen-Mc

This comment has been minimized.

Show comment
Hide comment
@Stephen-Mc

Stephen-Mc Aug 10, 2017

Collaborator

It indeed appears that a recent version of NUnit has introduced a change that breaks tests with SpecFlow+ Runner. We received the first reports of an issue with NUnit a few days ago, and have taken a look at a number of GitHub issues (including 2267, which you posted).

From what we have been able to determine, this will require a fix to the code in SpecFlow+ Runner in order to execute tests with more recent versions of NUnit. Until we have fixed this issue, we therefore recommend sticking with NUnit 3.6 instead of using 3.7 if you are using NUnit's Assert. We have already received confirmation from another user that downgrading NUnit did the trick for them.

Collaborator

Stephen-Mc commented Aug 10, 2017

It indeed appears that a recent version of NUnit has introduced a change that breaks tests with SpecFlow+ Runner. We received the first reports of an issue with NUnit a few days ago, and have taken a look at a number of GitHub issues (including 2267, which you posted).

From what we have been able to determine, this will require a fix to the code in SpecFlow+ Runner in order to execute tests with more recent versions of NUnit. Until we have fixed this issue, we therefore recommend sticking with NUnit 3.6 instead of using 3.7 if you are using NUnit's Assert. We have already received confirmation from another user that downgrading NUnit did the trick for them.

@shadowfoxish

This comment has been minimized.

Show comment
Hide comment
@shadowfoxish

shadowfoxish Aug 10, 2017

Thanks @Stephen-Mc, we'll keep on 3.6.1 until we hear there's a fix available.

shadowfoxish commented Aug 10, 2017

Thanks @Stephen-Mc, we'll keep on 3.6.1 until we hear there's a fix available.

@SabotageAndi

This comment has been minimized.

Show comment
Hide comment
@SabotageAndi

SabotageAndi Aug 14, 2017

Collaborator

We will close this issue, when it is fixed.

Collaborator

SabotageAndi commented Aug 14, 2017

We will close this issue, when it is fixed.

@SabotageAndi SabotageAndi reopened this Aug 14, 2017

@alexmaie

This comment has been minimized.

Show comment
Hide comment
@alexmaie

alexmaie Aug 18, 2017

Hey guys, we have a similiar issue with Specflow 2.2 and NUnuit 3.7.1 . We are getting the following exception.

Result StackTrace:
at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioStart(ScenarioInfo scenarioInfo)
at TechTalk.SpecFlow.TestRunner.OnScenarioStart(ScenarioInfo scenarioInfo)
at WebApi.Specs.Nunit.Sip.CTIControllerFeature.ScenarioSetup(ScenarioInfo scenarioInfo)
at WebApi.Specs.Nunit.Sip.CTIControllerFeature.IfACallHasEndendThenTheCallHistoryLogNeedsToBeUpdated(String username, String callId, String[] exampleTags) in
--TearDown
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.GetHookContainer(HookType hookType)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.InvokeHook(IBindingInvoker invoker, IHookBinding hookBinding, HookType hookType)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.FireEvents(HookType hookType)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.FireScenarioEvents(HookType bindingEvent)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd()
at TechTalk.SpecFlow.TestRunner.OnScenarioEnd()
at WebApi.Specs.Nunit.Sip.CTIControllerFeature.ScenarioTearDown()
Result Message:
System.NullReferenceException : Object reference not set to an instance of an object.
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.

This exception appears when running tests in parallel in VS and with the nunit console runner. If the tests are not run in parallel, everything works

alexmaie commented Aug 18, 2017

Hey guys, we have a similiar issue with Specflow 2.2 and NUnuit 3.7.1 . We are getting the following exception.

Result StackTrace:
at TechTalk.SpecFlow.Infrastructure.ContextManager.InitializeScenarioContext(ScenarioInfo scenarioInfo)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioStart(ScenarioInfo scenarioInfo)
at TechTalk.SpecFlow.TestRunner.OnScenarioStart(ScenarioInfo scenarioInfo)
at WebApi.Specs.Nunit.Sip.CTIControllerFeature.ScenarioSetup(ScenarioInfo scenarioInfo)
at WebApi.Specs.Nunit.Sip.CTIControllerFeature.IfACallHasEndendThenTheCallHistoryLogNeedsToBeUpdated(String username, String callId, String[] exampleTags) in
--TearDown
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.GetHookContainer(HookType hookType)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.InvokeHook(IBindingInvoker invoker, IHookBinding hookBinding, HookType hookType)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.FireEvents(HookType hookType)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.FireScenarioEvents(HookType bindingEvent)
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnScenarioEnd()
at TechTalk.SpecFlow.TestRunner.OnScenarioEnd()
at WebApi.Specs.Nunit.Sip.CTIControllerFeature.ScenarioTearDown()
Result Message:
System.NullReferenceException : Object reference not set to an instance of an object.
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.

This exception appears when running tests in parallel in VS and with the nunit console runner. If the tests are not run in parallel, everything works

@SabotageAndi

This comment has been minimized.

Show comment
Hide comment
@SabotageAndi

SabotageAndi Aug 29, 2017

Collaborator

With the release of NUnit 3.8 this is fixed on NUnit side.
Issue from NUnit: nunit/nunit#2242

Collaborator

SabotageAndi commented Aug 29, 2017

With the release of NUnit 3.8 this is fixed on NUnit side.
Issue from NUnit: nunit/nunit#2242

@yonashamailov

This comment has been minimized.

Show comment
Hide comment
@yonashamailov

yonashamailov Jun 19, 2018

Issue: cannot uninstall-package, receive object reference not set to an instance of an object
scenario :
in project 1 Selenium v=3.12.1, and other projects in solution have v ="3.5.1". (nunit 3.10.1)

->so I changed selenium version in all projects in the solution. (to same V3.5.1)
now
Project 1
"NUnit" version="3.5.0"
"Selenium.Support" version="3.5.1" (same version as in other projects)

I also had run time exceptions which was also solved after downgrade
Could not load type 'OpenQA.Selenium.Support.UI.WebDriverWait' from assembly 'WebDriver.Support, Version=3.12.1.0, Culture=neutral, PublicKeyToken=null'.
Source=project2

I use Vs2015, c#, selenium webdriver, nunit, nuget

yonashamailov commented Jun 19, 2018

Issue: cannot uninstall-package, receive object reference not set to an instance of an object
scenario :
in project 1 Selenium v=3.12.1, and other projects in solution have v ="3.5.1". (nunit 3.10.1)

->so I changed selenium version in all projects in the solution. (to same V3.5.1)
now
Project 1
"NUnit" version="3.5.0"
"Selenium.Support" version="3.5.1" (same version as in other projects)

I also had run time exceptions which was also solved after downgrade
Could not load type 'OpenQA.Selenium.Support.UI.WebDriverWait' from assembly 'WebDriver.Support, Version=3.12.1.0, Culture=neutral, PublicKeyToken=null'.
Source=project2

I use Vs2015, c#, selenium webdriver, nunit, nuget

@SabotageAndi

This comment has been minimized.

Show comment
Hide comment
@SabotageAndi

SabotageAndi Jun 20, 2018

Collaborator

@yonashamailov This is issue was closed nearly a year ago. Please open a new one.

Collaborator

SabotageAndi commented Jun 20, 2018

@yonashamailov This is issue was closed nearly a year ago. Please open a new one.

@yonashamailov

This comment has been minimized.

Show comment
Hide comment
@yonashamailov

yonashamailov Jun 20, 2018

@SabotageAndi
I just want to share my practice in similar issue:

I created new project for selenium test in c# (vs2015) in existing working solution (the rest old projects use selenium "3.5.1")
installed in new P selenium v=3.12.1, nunit 3.10.1 with nuget

nunit test had excepion (issue 1):
Could not load type 'OpenQA.Selenium.Support.UI.WebDriverWait' from assembly 'WebDriver.Support, Version=3.12.1.0, Culture=neutral, PublicKeyToken=null'.
Source="project2"

when I tried to uninstall this nunit version 3.10.1 (and install another version, some version close to this v) by nuget it failed:
"object reference not set to an instance of an object" - issue #2

->so I solved nunit/nuget issue (#2) by downgrading nunit to 3.5.0, & downgrading selenium to 3.5.1 ,(same version as in other projects) , nuget allow to do so

issue #1 was also solved after those downgrades

I use Vs2015, c#, selenium webdriver, nunit, nuget

yonashamailov commented Jun 20, 2018

@SabotageAndi
I just want to share my practice in similar issue:

I created new project for selenium test in c# (vs2015) in existing working solution (the rest old projects use selenium "3.5.1")
installed in new P selenium v=3.12.1, nunit 3.10.1 with nuget

nunit test had excepion (issue 1):
Could not load type 'OpenQA.Selenium.Support.UI.WebDriverWait' from assembly 'WebDriver.Support, Version=3.12.1.0, Culture=neutral, PublicKeyToken=null'.
Source="project2"

when I tried to uninstall this nunit version 3.10.1 (and install another version, some version close to this v) by nuget it failed:
"object reference not set to an instance of an object" - issue #2

->so I solved nunit/nuget issue (#2) by downgrading nunit to 3.5.0, & downgrading selenium to 3.5.1 ,(same version as in other projects) , nuget allow to do so

issue #1 was also solved after those downgrades

I use Vs2015, c#, selenium webdriver, nunit, nuget

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment