Fixes #916 - Wrap remote calls

[rprouse]

I managed to reproduce the issue by crashing the agent with a stack overflow in a test.

This fix wraps all of the remoting calls in ProcessRunner in a try/catch. If it crashes running or exploring tests, it returns a TestEngineResult that contains the remoting exception. The full results are now properly displayed. As a side effect, I noticed that while testing this issue, nunit-agents were left running in the background. This is no longer happening, I assume because we are now stopping the agents that are still running.

My only concern with this is the fake test results where I fake data I don't know. I checked the final TestResult.xml and it looks fine.

My now successfully reported crash run is below. Before, it ended with an exception and no results were displayed.

NUnit Console Runner 3.0.5778 (Debug)
Copyright (C) 2014 Charlie Poole

Runtime Environment
   OS Version: Microsoft Windows NT 6.3.9600.0
  CLR Version: 4.0.30319.42000

Test Files
    D:\Src\Spikes\ErrorNunit\ErrorNunit\bin\Debug\ErrorNunit.dll
    D:\Src\Spikes\CrashNunit\CrashNunit\bin\Debug\CrashNunit.dll


Process is terminated due to StackOverflowException.

Errors and Failures

1) Error : ErrorNunit.ErrorTest.ThrowsException
System.ArgumentException : This is totally wrong dude!
   at ErrorNunit.ErrorTest.ThrowsException() in D:\Src\Spikes\ErrorNunit\ErrorNunit\Class1.cs:line 19

2) Error : D:\Src\Spikes\CrashNunit\CrashNunit\bin\Debug\CrashNunit.dll
An existing connection was forcibly closed by the remote host

Server stack trace:
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter) in D:\Src\nunit\nunit\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 158

Run Settings
    ProcessModel: Multiple
    WorkDirectory: D:\src\nunit\nunit
    DomainUsage: Single
    NumberOfTestWorkers: 8

Test Run Summary
    Overall result: Failed
   Tests run: 1, Passed: 0, Errors: 1, Failures: 0, Inconclusive: 0
     Not run: 0, Invalid: 0, Ignored: 0, Explicit: 0, Skipped: 0
  Start time: 2015-10-27 19:48:52Z
    End time: 2015-10-27 19:49:00Z
    Duration: 7.196 seconds

The TestResult.xml is

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-run id="2" testcasecount="2" result="Failed" label="Error" total="2" passed="0" failed="2" inconclusive="0" skipped="0" asserts="0" start-time="2015-10-27 19:48:52Z" end-time="2015-10-27 19:49:00Z" duration="7.195691">
  <environment nunit-version="3.0.5778.28433" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.3.9600.0" platform="Win32NT" cwd="D:\src\nunit\nunit" machine-name="NAOU16869" user="Rob.Prouse" user-domain="BENTLEY" culture="en-CA" uiculture="en-US" os-architecture="x64" />
  <command-line><![CDATA["D:\src\nunit\nunit\bin\Debug\nunit-console.exe" --process=Multiple D:\Src\Spikes\ErrorNunit\ErrorNunit\bin\Debug\ErrorNunit.dll D:\Src\Spikes\CrashNunit\CrashNunit\bin\Debug\CrashNunit.dll]]></command-line>
  <settings>
    <setting name="ProcessModel" value="Multiple" />
    <setting name="WorkDirectory" value="D:\src\nunit\nunit" />
    <setting name="DomainUsage" value="Single" />
    <setting name="NumberOfTestWorkers" value="8" />
  </settings>
  <test-suite type="Assembly" id="1-1002" name="ErrorNunit.dll" fullname="D:\Src\Spikes\ErrorNunit\ErrorNunit\bin\Debug\ErrorNunit.dll" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2015-10-27 19:48:52Z" end-time="2015-10-27 19:48:52Z" duration="0.021682" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
    <properties>
      <property name="_PID" value="17160" />
      <property name="_APPDOMAIN" value="test-domain-ErrorNunit.dll" />
    </properties>
    <failure>
      <message><![CDATA[One or more child tests had errors]]></message>
    </failure>
    <test-suite type="TestSuite" id="1-1003" name="ErrorNunit" fullname="ErrorNunit" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2015-10-27 19:48:52Z" end-time="2015-10-27 19:48:52Z" duration="0.015634" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
      <failure>
        <message><![CDATA[One or more child tests had errors]]></message>
      </failure>
      <test-suite type="TestFixture" id="1-1000" name="ErrorTest" fullname="ErrorNunit.ErrorTest" classname="ErrorNunit.ErrorTest" runstate="Runnable" testcasecount="1" result="Failed" site="Child" start-time="2015-10-27 19:48:52Z" end-time="2015-10-27 19:48:52Z" duration="0.014228" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
        <failure>
          <message><![CDATA[One or more child tests had errors]]></message>
        </failure>
        <test-case id="1-1001" name="ThrowsException" fullname="ErrorNunit.ErrorTest.ThrowsException" methodname="ThrowsException" classname="ErrorNunit.ErrorTest" runstate="Runnable" seed="798259755" result="Failed" label="Error" start-time="2015-10-27 19:48:52Z" end-time="2015-10-27 19:48:52Z" duration="0.007142" asserts="0">
          <failure>
            <message><![CDATA[System.ArgumentException : This is totally wrong dude!]]></message>
            <stack-trace><![CDATA[   at ErrorNunit.ErrorTest.ThrowsException() in D:\Src\Spikes\ErrorNunit\ErrorNunit\Class1.cs:line 19]]></stack-trace>
          </failure>
        </test-case>
      </test-suite>
    </test-suite>
  </test-suite>
  <test-suite type="Assembly" id="2" name="CrashNunit.dll" fullname="D:\Src\Spikes\CrashNunit\CrashNunit\bin\Debug\CrashNunit.dll" runstate="NotRunnable" testcasecount="1" result="Failed" label="Error" start-time="2015-10-27 19:49:00Z" end-time="2015-10-27 19:49:00Z" duration="0.001" total="1" passed="0" failed="1" inconclusive="0" skipped="0" asserts="0">
    <failure>
      <message><![CDATA[An existing connection was forcibly closed by the remote host]]></message>
      <stack-trace><![CDATA[
Server stack trace: 
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter) in D:\Src\nunit\nunit\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 158]]></stack-trace>
    </failure>
  </test-suite>
</test-run>

[CharliePoole]

I feel like we may want to put this elsewhere at some point, or perhaps combine it with what we do in other cases. Good for the release though.

[CharliePoole]

Looks good to me! I too am a little nervous about the "fake" result. But we do this in other places as well. As I commented in the code, I think we will want to consolidate all our "fakery" somewhere, sometime.