NUnit3TestAdapter errors when running xUnit tests

[damonbarry]

My .NET Core codebase has a mix of xUnit and NUnit tests which all get built into the same directory with something like:

dotnet build -o c:\some\shared\dir .\my.sln

When I try to run some xUnit tests with:

dotnet vstest c:\some\shared\dir\xunit-test.dll

...the tests pass but I see an error in the console:

Error initializing RunSettings. Default settings will be used
System.IO.FileNotFoundException: Could not load file or assembly 'System.Xml.XPath.XmlDocument, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
File name: 'System.Xml.XPath.XmlDocument, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at NUnit.VisualStudio.TestAdapter.AdapterSettings.Load(String settingsXml)
   at NUnit.VisualStudio.TestAdapter.NUnitTestAdapter.Initialize(IDiscoveryContext context, IMessageLogger messageLogger) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestAdapter.cs:line 126


Exception System.IO.FileNotFoundException, Exception thrown executing tests
Could not load file or assembly 'System.Xml.XPath.XmlDocument, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter)
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 99

It seems that NUnit's test adapter is selected first and fails, so dotnet test moves on to the xUnit adapter, which runs the tests. Note that I don't see this error if:

• I use dotnet publish instead of dotnet build (System.Xml.XPath.XmlDocument.dll is copied into the target directory)
• I use dotnet test instead of dotnet vstest

The error is similar to what's described in #589, but the repro seems very different so I didn't want to muddy that issue.

Environment

NUnit 3.11.0
NUnit3TestAdapter 3.11.0
.NET Core SDK 2.2.202
Windows 10.0.17763 (RS5)

Repro (from PowerShell)

mkdir ~\projects\xunit-nunit-clash
cd ~\projects\xunit-nunit-clash
dotnet new xunit -n xunit-test
dotnet new nunit -n nunit-test
dotnet build .\xunit-test\  -f netcoreapp2.2 -o ..\bin\
dotnet build .\nunit-test\  -f netcoreapp2.2 -o ..\bin\
dotnet vstest .\bin\xunit-test.dll

[OsirisTerje]

Can you please upload a repro project?
Make it as small as you can, it should compile "as is".

[damonbarry]

Sure, here you go. Unzip it, change into that directory, and run the last three commands from the repro steps in my original post:
xunit-nunit-clash.zip

[OsirisTerje]

Thanks!
And confirmed!

I see the nunit tests are running as they should when nunit-test.dll is used, but crashes as you describe when the xunit-test assembly is run. The VSTest is searching for all adapters in a folder, so I assume it tries to run both. Why it should fail on the nunit one when running the xunit-tests are very strange.

[OsirisTerje]

What happens is that the NUnit adapter is running under .net core 1 and not 2.2.
The error message is:
[6/26/2019 10:55:35 AM Warning] Test run will use DLL(s) built for framework .NETCoreApp,Version=v1.0 and platform X86. Following DLL(s) do not match framework/platform settings.
nunit-test.dll is built for Framework 2.2 and Platform AnyCPU.

@jnm2 @rprouse Comments ?

[jnm2]

This part doesn't make sense to me unless you override VSTest's autodetection by passing a /Framework: parameter:

Test run will use DLL(s) built for framework .NETCoreApp,Version=v1.0 and platform X86. Following DLL(s) do not match framework/platform settings.

Can we confirm for sure that VSTest is choosing .NET Core 1.0 solely because of our adapter in all circumstances, or is there another factor?

[rprouse]

A possible hacky workaround might be to add a reference to NUnit to the xUnit tests and add the NonTestAssemblyAttribute to the xUnit test suites so that NUnit ignores them.

As for what is causing this, I can't think of anything.

[OsirisTerje]

Note: If you run the nunit-test.dll, all works as it should. This happens when we run the xunit-test.dll. Since they are in the same folder, it sorts of picks up the nunit too.
Btw: The error message I show above comes from Visual Studio discovery phase, and clearly shows that we somehow are forced into netcore 1. I tried using the framework parameter, but it didnt have any effect on this.

I've added up the repro code (thanks @damonbarry) with a sln file , it is named BlankSln.sln, but containts the 2 projects. Then it is easy to see the discovery phase too, and the error message. Repo is here: https://github.com/OsirisTerje/Issue629

Suspicious when we reference netcore 1.0, and then this pops up.

[OsirisTerje]

@damonbarry
Can you try this version :
Install-Package NUnit3TestAdapter -Version 3.13.0-dev-01090 -Source https://www.myget.org/F/nunit/api/v3/index.json

This version includes the net core 2.0, and it works on my machine.

[damonbarry]

Yep, that worked for me locally on my simple repro case. 👍
I'll try it later on my real codebase. Thanks!

[damonbarry]

Noticed the release this morning...thanks!

[OsirisTerje]

@damonbarry Just curious: Through what did you notice the release? I'm trying to publish as many places as possible, so is just curious how people pick it up.

[damonbarry]

Not through any sort of automatic notification, unfortunately. You added this issue to your 3.14 milestone, which I would check periodically. Once a date was attached to the milestone, I checked back daily until this morning.

[OsirisTerje]

Ahh... I see. I'm trying to get the dates correct, but I got a few days delayed here due to travels during the summer vacation. Normally a good time for releasing stuff, but not this year ;-)

Hope it works for your production code :-)