Spurious report of 'test with the same name' in solutions with multiple projects

[idg10]

If I have two test projects both with the dotnet-test-nunit package installed, test discovery goes wrong - I get reports of "A test with the same name '..' already exists" even though the test name is in fact unique.

This only affects discovery - I can run all tests without problem, although when one of the affects tests fails, Test Explorer misattributes the failure to a different test. I can work around the problem by unloading projects so that only the one I'm working in right now is loaded, but this is not ideal.

I encountered this problem using .xproj projects compiling for the full .NET Framework. (I'm writing an ASP.NET Core 1.0 application, so I need to use .xproj projects, but I'm not targeting .NET Core.) I don't know if the same problem applies to projects that target .NET Core, but full framework .xproj projects are critically important for me since we use ASP.NET Core but require full .NET Framework features.

Steps to reproduce in VS 2015 with ASP.NET Core 1.0 preview2 tooling:

1. Create a new ASP.NET Core Web Application (.NET Framework) called WebApp

2. Add a .NET Class Library (.NET Core) called TestWebApp

3. In TestWebApp Modify frameworks section of project.json to use .NET Framework 4.61:

   "frameworks": {
   "net461": { }
   }

4. In TestWebApp remove "NETStandard.Library": "1.6.0" from dependencies section of `project.json.

5. Add reference from TestWebApp to WebApp

6. Add NuGet packages NUnit (3.4.1) and dotnet-test-nunit (3.4.0-beta-2) to TestWebApp

7. In TestWebApp add "testRunner": "nunit" to `project.json

8. Add a test method to a class in TestWebApp

9. Build solution. Verify that the test appears in the Test Explorer. (Might need to quit and restart Visual Studio at this point.) So far so good.

10. Add a .NET Class Library (.NET Core) called Test2.

11. Make the same modifications to project.json as with TestWebApp to make this use the full .NET Framework.

12. Install the same NUnit and dotnet-test-nunit packages you installed in TestWebApp in Test2.

13. In Test2 add "testRunner": "nunit" to `project.json

14. Write a class with one test in Test2 with different class and method names than the one in TestWebApp

15. Rebuild. At this stage, I found that it was only looking for tests in TestWebApp and I had to quit and restart Visual Studio to get it to look for tests in both test libraries.

16. Note that only one of the tests shows up in Test Explorer. (For me, it's the one from the Test2 project). There's a surprising error in the Tests section of the Output panel:

------ Discover test started ------
========== Discover test finished: 0 found (0:00:00.095533) ==========
------ Discover test started ------
Discovering tests in 'C:\Users\Ian\Documents\Visual Studio 2015\Projects\WebApp\src\Test2\project.json' ["C:\Program Files\dotnet\dotnet.exe" test "C:\Users\Ian\Documents\Visual Studio 2015\Projects\WebApp\src\Test2\project.json" --output "C:\Users\Ian\Documents\Visual Studio 2015\Projects\WebApp\src\Test2\bin\Debug\net461\win7-x64" --port 12258 --parentProcessId 21036 --no-build]
Discovering tests in 'C:\Users\Ian\Documents\Visual Studio 2015\Projects\WebApp\src\TestWebApp\project.json' ["C:\Program Files\dotnet\dotnet.exe" test "C:\Users\Ian\Documents\Visual Studio 2015\Projects\WebApp\src\TestWebApp\project.json" --output "C:\Users\Ian\Documents\Visual Studio 2015\Projects\WebApp\src\TestWebApp\bin\Debug\net461\win7-x64" --port 12263 --parentProcessId 21036 --no-build]
A test with the same name 'TestWebApp.Class1.Test' already exists. This test is not added to the test window.
========== Discover test finished: 2 found (0:00:02.1412771) ==========

Notice the spurious report of a test with the same name as another that already exists.

I've attached a solution that exhibits this problem: WebApp.zip

[idg10]

You can also get the repro at https://github.com/idg10/nunit-repro1

[rprouse]

Thanks, I will take a look when I get a chance. If any other team members have time to triage this, I would appreciate it.

[ghost]

If we search google with quoted string "already exists. This test is not added to the test window.", this seems like issue with nunit exclusively.
This issue has been reported in VS adapter repo in different forms: e.g. nunit/nunit-vs-adapter#82.

[CharliePoole]

The problem is well understood wrt the nunit VS adapter, but may not be one we want to fix.

The issue is that VS and NUnit itemize tests differently. The problem is well described by the authors of Fixie wrt their own adapter: https://github.com/fixie/fixie/blob/master/src/Fixie.VisualStudio.TestAdapter/README.md When I get a chance, I'll write something about this for our own docs.

In general, the problem only comes up with overloads of a test method or use of parameterized tests. In the case of parameterized tests, NUnit considers each set of arguments as a unique test while Visual Studio considers each of them as producing a new result for a single test. When I first wrote the adapter, the TestWindow guys at Microsoft suggested we change how NUnit looks at tests. We didn't. :-)

I haven't commented on this issue before because I don't have a clue whether any of this applies to the dotnet-test-nunit package. One of these days, I'm going to get familiar with it. :-)

It doesn't appear that this issue relates to either overloads or parameterized tests. OTOH, if the same test name is being returned to VS from two different assemblies, that could easily cause the problem. Another major difference between NUnit and VS is that VS expects test names to be unique. NUnit doesn't care. As an experiment, I once forced all my tests to have the same name. Everything ran perfectly in the console, except for the fact that the output didn't make much sense to me. :-)

Question: Is the dotnet-test-nunit package also an adapter? Or is it simply called by a microsoft-provided adapter?

[rprouse]

@CharliePoole it is hard to define the boundary between an adapter and something called by an adapter. There is no VSIX possible, but it is discovered if installed as a NuGet package and treated like an adapter. My understanding is that it is similar to the desktop framework NUnit Adapter in that VS runs a test process which communicates to our adapter. The difference is that dotnet-test-nunit is an executable in it's own right and communicates back to the test process via TCP, much like our agent and engine. It is further confused by the fact that this project can also run standalone as a console runner. The mode is determined by the command line arguments.

All that said, I have found that this adapter suffers from all the same issues as other adapters plus a few news ones because of the new interface/protocol.

[CharliePoole]

Well, from a certain perspective it's not hard at all. :-)

Of course the generic term "adapter" has a broader meaning, but a VS test adapter has always been an assembly that is loaded by the TestExplorer and which implements ITestDiscoverer and ITestExecutor at a minimum.

Is the interface you use documented by MS somewhere? Eventually I want to spend some time getting familiar with it. If similar problems are coming up, it leads me to suspect that we are doing the same things in different code, where maybe we don't have to.

[rprouse]

The interface is documented, but not well, I think I sent you the links when I started the project, but don't have them handy. I will need to dig them up again.

[CharliePoole]

Don't bother digging. If you sent it I'm sure I can fix find it.

[idg10]

To be clear, there are no name collisions in the tests here. (Not in the repro I've produced, nor in the real project in which I originally encountered this.) Nor am I using parameterized tests.

The problem in this case is that when I have two different projects I get errors reporting that two tests have the same name when in fact they don't. So I'm not sure this is just another instance of the other problem you're describing.

[ghost]

Yes, there are two test projects in that solution with different names, different class names and different method names -- and yet this error occurs as if there is a name collision.

[dougkpowers]

I was having the same problem and believe pull request #82 will fix it.