dotnet test in IDE mode fails during discovery of net451 tests and only attempts to run for the first defined target framework

[plioi]

Sample project/commit to reproduce this issue: plioi/parsley@513cc2f

As of that commit, I can run the tests in Test Explorer as expected. The Output pane shows all 115 tests discovered and executed within bin\Debug\netcoreapp1.0. At the command line, all 115 tests are correctly run twice, once for each of the two defined target frameworks.

At this point, I was surprised, though, that it wasn't also running another 115 for net451 when run in Test Explorer. From the documentation: "You can target both net4xx and netcoreapp simply by adding both frameworks together in your project.json file. When you run dotnet test with multiple framework entries, the system will run all your framework tests, one after another." That statement may only apply to the console mode.

As of the linked commit, my tests' project.json is this:

{
  "testRunner": "xunit",
  "dependencies": {
    "Parsley": { "target": "project" },
    "Shouldly": "2.8.0",
    "xunit": "2.2.0-beta2-build3300",
    "dotnet-test-xunit": "2.2.0-preview2-build1029"
  },

  "frameworks": {
    "netcoreapp1.0": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        }
      }
    },
    "net451": {
      "dependencies": {
        "Microsoft.NETCore.Platforms": "1.0.1"
      }
    }
  }
}

Since Test Explorer was only attempting to run this for netcoreapp1.0, I figured I was merely bumping into a limitation on the Microsoft side of the VS/xunit handshake, so I attempted to swap the order of the two framework descriptions in the project.json, so that the net451 section came first. The Output pane shows an attempt to discover within bin\Debug\net451 this time, but then fails:

------ Discover test started ------
========== Discover test finished: 0 found (0:00:01.0510601) ==========
------ Discover test started ------
Discovering tests in 'C:\dev\parsley\src\Parsley.Tests\project.json' ["C:\Program Files\dotnet\dotnet.exe" test "C:\dev\parsley\src\Parsley.Tests\project.json" --output "C:\dev\parsley\src\Parsley.Tests\bin\Debug\net451\win7-x64" --port 16221 --parentProcessId 5940 --no-build]
Unable to start C:\Program Files\dotnet\dotnet.exe
dotnet-test Error: 0 : [ReportingChannel]: Waiting for message failed System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.DotNet.Tools.Test.ReportingChannel.ReadMessages()

Unhandled Exception: System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.DotNet.Tools.Test.ReportingChannel.ReadMessages()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
dotnet-test Error: 0 : [ReportingChannel]: Error sending System.IO.IOException: Unable to write data to the transport connection: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, SocketError& errorCode)
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.Send(Message message)
dotnet-test Error: 0 : System.IO.IOException: Unable to write data to the transport connection: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, SocketError& errorCode)
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.Send(Message message)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.SendError(String error)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.SendError(Exception ex)
   at Microsoft.DotNet.Tools.Test.DesignTimeRunner.HandleDesignTimeMessages(ProjectContext projectContext, DotnetTestParams dotnetTestParams)
   at Microsoft.DotNet.Tools.Test.DesignTimeRunner.DoRunTests(ProjectContext projectContext, DotnetTestParams dotnetTestParams)
   at Microsoft.DotNet.Tools.Test.BaseDotnetTestRunner.RunTests(ProjectContext projectContext, DotnetTestParams dotnetTestParams, BuildWorkspace workspace)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)
========== Discover test finished: 0 found (0:00:01.0580605) ==========

I have two questions:

1. Does dotnet test in IDE mode mistakenly run only one defined framework, unlike when I run it at the command line, in which case I should create an issue to the dotnet CLI team? Or, is that part in xUnit's control?
2. Does the exception thrown look like an xUnit issue, or again a dotnet CLI issue?

[Romanx]

I was getting this error and it seemed to be that netcoreapp1.0 had to come before my net46 framework dependency. My project.json framework section now looks like:

"frameworks": {
        "netcoreapp1.0": {
            "dependencies": {
                "Microsoft.NETCore.App": {
                    "type": "platform",
                    "version": "1.0.0"
                }
            }
        },
        "net46": {
            "dependencies": {
                "Microsoft.NETCore.Platforms": "1.0.1"
            }
        }
    }

[plioi]

@Romanx But, with that change, is Test Explorer able to run your tests for both netcoreapp1.0 and net46?

[bradwilson]

The limitation of running just the first framework is out of our control, unfortunately. The Microsoft team responsible for this is aware of this current limitation.

[bradwilson]

The socket issue is something that doesn't look like us, and should probably be reported to Microsoft.

[masaeedu]

@bradwilson Could the "first framework-only" limitation be documented under the "Targeting desktop .NET" section on this page please?

[rclabo]

My situation seems very similar and I could use some help. Here is my project.json for my Test library:

{
  "testRunner": "xunit",
  "version": "1.0.0-*",

   "dependencies": {
    "wwwGiftOasisResponsive": "1.0.0-*",
    "xunit": "2.1.0",
    "xunit.runner.visualstudio": "2.1.0"
  },

  "frameworks": {
     "net461": {
      "dependencies": {
        "MP_Prims": {
              "target": "project"
         }
     }
    }
  }
}

And here are the exceptions produced in the output window of VS2015 when trying to run tests:

------ Discover test started ------
Discovering tests in 'C:\Users\Ron Clabo\Documents\Visual Studio           2015\Projects\wwwGiftOasisResponsive\test\project.json' ["C:\Program Files\dotnet\dotnet.exe" test     "C:\Users\Ron Clabo\Documents\Visual Studio     2015\Projects\wwwGiftOasisResponsive\test\project.json" --output "C:\Users\Ron Clabo\Documents\Visual Studio 2015\Projects\wwwGiftOasisResponsive\test\bin\Debug\net461\win7-x64" --port 50330 --parentProcessId 8540 --no-build]
Unable to start C:\Program Files\dotnet\dotnet.exe
dotnet-test Error: 0 : [ReportingChannel]: Waiting for message failed System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.DotNet.Tools.Test.ReportingChannel.ReadMessages()

Unhandled Exception: System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. --->    System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.Stream.ReadByte()
   at System.IO.BinaryReader.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.DotNet.Tools.Test.ReportingChannel.ReadMessages()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
dotnet-test Error: 0 : [ReportingChannel]: Error sending System.IO.IOException: Unable to write data to the transport connection: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.. ---> System.ObjectDisposedException: Cannot access a   disposed object.
  Object name: 'System.Net.Sockets.Socket'.
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, SocketError& errorCode)
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.Send(Message message)
dotnet-test Error: 0 : System.IO.IOException: Unable to write data to the transport connection:     Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.Socket'.
  at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags,  SocketError& errorCode)
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
   at System.IO.BinaryWriter.Write(String value)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.Send(Message message)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.SendError(String error)
   at Microsoft.DotNet.Tools.Test.ReportingChannel.SendError(Exception ex)
   at Microsoft.DotNet.Tools.Test.DesignTimeRunner.HandleDesignTimeMessages(ProjectContext   projectContext, DotnetTestParams dotnetTestParams)  
   at Microsoft.DotNet.Tools.Test.DesignTimeRunner.DoRunTests(ProjectContext projectContext, DotnetTestParams dotnetTestParams)
   at Microsoft.DotNet.Tools.Test.BaseDotnetTestRunner.RunTests(ProjectContext projectContext, DotnetTestParams dotnetTestParams, BuildWorkspace workspace)
   at Microsoft.DotNet.Tools.Test.TestCommand.DoRun(String[] args)
========== Discover test finished: 0 found (0:00:00.7380422) ==========

I'm trying to use this xproj library to hold my xunit tests which I want to run against the following two projects: wwwGiftOasisResponsive Asp.Net Core Project (.Net Framework) project that targets Net461 and MP_Prims which is a Windows class library which targets framework 4.61.

Questions
Anyone know how to resolve this exception and get my xproj library to run tests against a Asp.Net Core Project (.Net Framework) project that targets Net461 and a Windows class library which targets framework 4.61?

[rclabo]

I found a solution to my issue and am posting it here in case it helps someone. What I have learned is that using an xproj library targeting net461 to test an Asp.Net Core Project (.Net Framework) project targeting net461 and a a Windows class library targeting framework 4.61 works but the setup seems to be very finicky and fragile. The key insight for me came from this thread aspnet/Tooling#245 where @BradRem indicated that the folder structure of the projects seemed to the source of issues.

Initially I tried unsuccessfully to use this folder structure:

src
    ____Asp.Net Core Project (.Net Framework) project targeting Net461
    ____Windows class library targeting framework 4.61

test
    ____Core Library used to run xUnit Tests

But when I tried to run the tests using this folder structure it produced the exception I previously posted.

But when I changed to the following folder structure I was able to get it to work:

src
    ____Asp.Net Core Project (.Net Framework) project targeting Net461
    ____Windows class library targeting framework 4.61
    ___Core Library used to run xUnit Tests

So the key was putting the folder that housed the testing class library in the src folder where the other projects folders were.

That said, the other thing that seemed to make a big difference was to add the references to the other two projects to the Core Library test project at the same time rather than one at a time.

[msgallagher]

this thread might also help: https://github.com/dotnet/cli/issues/4784

[bradwilson]

Closed for age.