Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests are duplicated/doubled up in Visual Studio Test Explorer #900

Closed
realfaisalk opened this Issue Jul 19, 2017 · 19 comments

Comments

Projects
None yet
4 participants
@realfaisalk
Copy link

commented Jul 19, 2017

Windows 10 version - 1703, build 15063.138
Visual Studio 2017 Enterprise - 15.2
Specflow v2.20
SpecFlow.MsTest v2.2.0
SpecRun.Runner v1.6.0
SpecRun.SpecFlow v1.6.0
SpecRun.SpecFlow.2-2-0 v1.6.0

Hi,

This is not a blocker, but since upgrading SpecFlow from 2.1.0 to 2.2.0 and SpecRun 1.5.2 to 1.6.0 I've noticed that when tests are grouped by "Traits" in Test Explorer they are duplicated/doubled up (see attached screenshot).
I think it may be due to having both SpecRun.SpecFlow v1.6.0 & SpecRun.SpecFlow.2-2-0 v1.6.0, but they seem to have been added by NuGet automatically.
Do I need both? which is the "correct" one to have?

Thanks
duplicates_in_test explorer

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Jul 20, 2017

What do you have configured as unitTestProvider in you app.config?

@realfaisalk

This comment has been minimized.

Copy link
Author

commented Jul 20, 2017

Hi Andi,

Just 'SpecRun'

@PiotrZienkowicz

This comment has been minimized.

Copy link

commented Jul 24, 2017

I have problems with same bug:
test explorer

Visual Studio 2015 Community Edition + SpecFlow For VS2015 (v2017.1.6)

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Jul 25, 2017

@realfaisalk
Do you have this behavior in every project or only in this one?
Could you check your packages folder, if there are multiple versions of the SpecRun.Runner package.

@PiotrZienkowicz

This comment has been minimized.

Copy link

commented Jul 25, 2017

@SabotageAndi
I have this problem in diffrent projects and only one SpecRun.Runner.1.6.0.

@realfaisalk

This comment has been minimized.

Copy link
Author

commented Jul 25, 2017

@SabotageAndi

I have 2 projects in 1 solution. Both projects have this problems.

Packages:
image

@teknlgyuser

This comment has been minimized.

Copy link

commented Jul 31, 2017

@SabotageAndi

I have this problem as well.

The traits display fine until I upgrade the packages for the project to these:

package id="SpecFlow" version="2.2.0" targetFramework="net451"
package id="SpecRun.Runner" version="1.6.0" targetFramework="net451"
package id="SpecRun.SpecFlow" version="1.6.0" targetFramework="net451"
package id="SpecRun.SpecFlow.2-2-0" version="1.6.0" targetFramework="net451"

app.config:

specFlow>
unitTestProvider name="SpecRun"
plugins
add name="SpecRun"
plugins
specFlow

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Aug 3, 2017

Could you try one of the examples from https://github.com/techtalk/SpecFlow.Plus.Examples and see if there you also have duplicated tests?

@realfaisalk

This comment has been minimized.

Copy link
Author

commented Aug 3, 2017

Hi,

Have tried with "CustomDeploymentSteps --> CalculatorFeature".
With the 'old' versions of Specflow & SpecRun the tests show fine in Test Explorer, including tags:
image

However when I upgrade to the latest versions then with unitTestProvider 'SpecRun' no tests are being shown in Test Explorer.

If I change unitTestProvider to 'MsTest' then they are being shown:
image

I get this sometimes. I have to "faff" around with the app.config file a few times before 'SpecRun' "sticks" and tests are shown in Test Explorer:
image

Tests are duplicated with versions 2.2.0 & 1.6.0

@SabotageAndi SabotageAndi self-assigned this Aug 8, 2017

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Aug 8, 2017

I can reproduce the behaviour and will have a look at it.

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Aug 9, 2017

I found the error and I am working on a fix.

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Aug 22, 2017

This is fixed and will be released with the next release of the runner.

@teknlgyuser

This comment has been minimized.

Copy link

commented Aug 22, 2017

Great. Thanks for working on this. When do you anticipate the next release?

@realfaisalk

This comment has been minimized.

Copy link
Author

commented Aug 22, 2017

Thanks Andi,

Will test soon after the next release.

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Aug 22, 2017

@teknlgyuser probably in the next weeks

@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Sep 12, 2017

SpecFlow+Runner 2.2.1 is released which include the fix for this issue.

@realfaisalk

This comment has been minimized.

Copy link
Author

commented Sep 22, 2017

Hi,

Just got round to updating the NuGet packages, but am getting the following error when I try to use the Specflow UnitTestProvider:

[22/09/2017 09:31:52 Informational] SpecFlow + Runner Discovery started
[22/09/2017 09:31:52 Informational] SpecRun Evaluation Mode: Please purchase at http://www.specflow.org/plus to remove test execution delay.
[22/09/2017 09:31:52 Informational] Discovering target: Default
[22/09/2017 09:31:52 Informational] Discovering tests from assembly: C:\Users\XXXX\Source\Workspaces\Automation Framework\src\02 Dev\WebAutomation\bin\Debug\WebAutomation.dll
[22/09/2017 09:31:52 Informational] Error for test assembly C:\Users\XXXX\Source\Workspaces\Automation Framework\src\02 Dev\WebAutomation\bin\Debug\WebAutomation.dll
[22/09/2017 09:31:52 Informational] System.Runtime.Serialization.SerializationException: Type is not resolved for member 'TechTalk.SpecRun.Framework.Remoting.RemoteDomainInitializer,TechTalk.SpecRun.Framework.Utils, Version=1.6.0.0, Culture=neutral, PublicKeyToken=null'.
   at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName)
   at TechTalk.SpecRun.Framework.Remoting.AppDomainReference.CreateObject[T]()
   at TechTalk.SpecRun.Framework.Remoting.AppDomainManager.CreateAppDomainFor(String assemblyPath, Boolean useConfig, String customConfigFilePath)
   at TechTalk.SpecRun.Framework.TestSuiteProviders.TestSuiteProvider.BuildTestCollectionFromAssembly(String testAssemblyPath, FilterItem filterPredicate, INodeIdProvider& nodeIdProvider, TestCollection parentCollection)
   at TechTalk.SpecRun.Framework.TestSuiteProviders.TestSuiteProvider.GetTestSuite(TestProfile testProfile)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestDiscoverer.DiscoverTests(String source, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestDiscoverer.DiscoverTests(IEnumerable`1 sources, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)
[22/09/2017 09:31:52 Error] An exception occurred while test discoverer 'SpecRunTestDiscoverer' was loading tests. Exception: Type is not resolved for member 'TechTalk.SpecRun.Framework.Remoting.RemoteDomainInitializer,TechTalk.SpecRun.Framework.Utils, Version=1.6.0.0, Culture=neutral, PublicKeyToken=null'.
[22/09/2017 09:31:53 Informational] ========== Discover test finished: 183 found (0:00:00.5752035) ==========

Seems to have some reference to v1.6.0 stuck somewhere?

Also no tests are shown in Test Explorer if I just use Specflow, but if I use the MSTest UnitTestProvder then tests are shown.

I have followed the steps from here https://specflow.org/documentation/Troubleshooting-Visual-Studio-Integration/ but it has not fixed it

Note: I have 3 Specflow projects under the same solution.

One of the packages.config file:

<package id="ExtentReports" version="3.0.2" targetFramework="net452" />
 <package id="MSTest.TestAdapter" version="1.1.18" targetFramework="net452" />
 <package id="MSTest.TestFramework" version="1.1.18" targetFramework="net452" />
 <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net452" />
 <package id="NUnit" version="3.6.1" targetFramework="net452" />
 <package id="Selenium.Firefox.WebDriver" version="0.18.0" targetFramework="net452" />
 <package id="Selenium.Support" version="3.4.0" targetFramework="net452" />
 <package id="Selenium.WebDriver" version="3.4.0" targetFramework="net452" />
 <package id="Selenium.WebDriver.ChromeDriver" version="2.31.0" targetFramework="net452" />
 <package id="Selenium.WebDriver.IEDriver" version="3.4.0" targetFramework="net452" />
 <package id="Selenium.WebDriver.MicrosoftDriver" version="15.15063.0" targetFramework="net452" />
 <package id="SpecFlow" version="2.2.1" targetFramework="net452" />
 <package id="SpecFlow.MsTest" version="2.2.1" targetFramework="net452" />
 <package id="SpecRun.Runner" version="1.6.1" targetFramework="net452" />
 <package id="SpecRun.SpecFlow" version="1.6.1" targetFramework="net452" />
 <package id="SpecRun.SpecFlow.2-2-0" version="1.6.1" targetFramework="net452" />
 <package id="System.ValueTuple" version="4.3.1" targetFramework="net452" />
@SabotageAndi

This comment has been minimized.

Copy link
Collaborator

commented Sep 22, 2017

@realfaisalk This has nothing todo with this issue.
Please open a new one.

@realfaisalk

This comment has been minimized.

Copy link
Author

commented Sep 22, 2017

Apologies, I thought it might be something to do with the fix from this issue, but it looks like it was just Visual Studio playing up. Had to completely remove my projects and re-download them from TFS.

**Can confirm that the duplication has gone now and tests are displayed correctly in Test Explorer.

Thanks Specflow team!**

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.