RemotingException when performing multiple runs in a single process

[yaakov-h]

I've come across a strange edge case with the NUnit engine.

As background info - I'm using the NUnit Engine APIs directly in order to integrate it into an in-house distributed CI system.

I have a particular assembly which uses TestContext.Current.TestDirectory inside a TestCaseSource - this was previously discussed at nunit/nunit#2036

When calling ITestRunner.RunTests(listener, filter), the engine behaves as expected. When calling it a second time, however, the result XML has zero tests run, zero passed, zero failed.

I've posted a repro case to https://github.com/yaakov-h/NUnitMultiRunRepro

If you build it and run:

> NUnitTestRunner\bin\Debug\NUnitTestRunner.exe NUnitGuineaPig\bin\Debug\NUnitGuineaPig.dll

the engine passes the test the first time, but the test case source failed on subsequent runs with a RemotingException due to some sort of IPC failure:

NUnitTestRunner Starting...
Using NUnit Engine 3.7.0.0
Created engine instance.
Will run tests on NUnitGuineaPig.dll (NUnit Framework 3.9.0.0)
[Iteration #1]: Starting...
[Iteration #1]: Created runner.
[Iteration #1]: Starting tests...
[Iteration #1]: Starting...
[Iteration #1]: Ran 5 tests.
[Iteration #1]: NUnitGuineaPig.TestClass.FileExists("nunit.framework.dll"): Passed
[Iteration #1]: NUnitGuineaPig.TestClass.FileExists("nunit.framework.xml"): Passed
[Iteration #1]: NUnitGuineaPig.TestClass.FileExists("NUnitGuineaPig.dll"): Passed
[Iteration #1]: NUnitGuineaPig.TestClass.FileExists("NUnitGuineaPig.pdb"): Passed
[Iteration #1]: NUnitGuineaPig.TestClass.FileExists("nunit_random_seed.tmp"): Passed
[Iteration #1]: Completed.
[Iteration #2]: Starting...
[Iteration #2]: Created runner.
[Iteration #2]: Starting tests...
[Iteration #2]: Starting...
[Iteration #2]: Ran 1 tests.
[Iteration #2]: NUnitGuineaPig.TestClass.FileExists: Failed
System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
  ----> System.Runtime.Remoting.RemotingException : This remoting proxy has no channel sink which means either the server has no registered server channels that are listening, or this application has no suitable client channel to talk to the server.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at NUnit.Framework.TestCaseSourceAttribute.GetTestCaseSource(IMethodInfo method)
   at NUnit.Framework.TestCaseSourceAttribute.GetTestCasesFor(IMethodInfo method)
--RemotingException
   at System.Runtime.Remoting.Proxies.RemotingProxy.InternalInvoke(IMethodCallMessage reqMcmMsg, Boolean useDispatchMessage, Int32 callType)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Framework.Internal.TestExecutionContext.get_CurrentTest()
   at NUnit.Framework.TestContext.get_TestDirectory()
   at NUnitGuineaPig.TestClass.get_FilesInCurrentDirectory() in C:\Temp\NUnitRepro\NUnitRepro\NUnitGuineaPig\TestClass.cs:line 18
[Iteration #2]: Completed.
[Iteration #3]: Starting...
<.... more of the above, failure, removed to keep this shorter>
[Iteration #5]: Completed.
NUnitTestRunner Finished. Disposing of engine...
Completed.

If I keep the ITestRunner alive and re-use it, then I don't see this issue, which would suggest that the engine isn't handling multiple runners correctly.

If you add a test to the test filter - for example:

builder.AddTest(@"NUnitGuineaPig.TestClass.FileExists(""NUnitGuineaPig.dll"")");`

then NUnit doesn't discover anything or print the above error. Instead, it produces the following XML:

<test-run id="2" name="NUnitGuineaPig.dll" fullname="C:\Temp\NUnitRepro\NUnitRepro\NUnitGuineaPig\bin\Debug\NUnitGuineaPig.dll" testcasecount="1" result="Passed" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.7.0.0" clr-version="4.0.30319.42000" start-time="2017-12-05 03:38:10Z" end-time="2017-12-05 03:38:10Z" duration="0.686774">
  <command-line><![CDATA["C:\Temp\NUnitRepro\NUnitRepro\NUnitTestRunner\bin\Debug\NUnitTestRunner.exe" ..\..\..\NUnitGuineaPig\bin\Debug\NUnitGuineaPig.dll]]></command-line>
  <filter>
    <test>NUnitGuineaPig.TestClass.FileExists("NUnitGuineaPig.dll")</test>
  </filter>
  <test-suite type="Assembly" id="1-1003" name="NUnitGuineaPig.dll" fullname="C:\Temp\NUnitRepro\NUnitRepro\NUnitGuineaPig\bin\Debug\NUnitGuineaPig.dll" runstate="Runnable" testcasecount="1" result="Passed" start-time="2017-12-05 03:38:10Z" end-time="2017-12-05 03:38:10Z" duration="0.011542" total="0" passed="0" failed="0" warnings="0" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.9.0.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 10.0.16299.0" platform="Win32NT" cwd="C:\Temp\NUnitRepro\NUnitRepro\NUnitTestRunner\bin\Debug" machine-name="REDACTED" user="REDACTED" user-domain="REDACTED" culture="en-AU" uiculture="en-US" os-architecture="x64" />
    <settings>
      <setting name="ImageRuntimeVersion" value="4.0.30319" />
      <setting name="ImageTargetFrameworkName" value=".NETFramework,Version=v4.7" />
      <setting name="ImageRequiresX86" value="False" />
      <setting name="ImageRequiresDefaultAppDomainAssemblyResolver" value="False" />
      <setting name="RuntimeFramework" value="net-4.0" />
      <setting name="NumberOfTestWorkers" value="20" />
    </settings>
    <properties>
      <property name="_PID" value="12104" />
      <property name="_APPDOMAIN" value="domain-90d4e6a1-NUnitGuineaPig.dll" />
    </properties>
  </test-suite>
</test-run>

[ChrisMaddock]

Thanks for the repro @yaakov-h.

I can reproduce this. Cutting out the custom runner functionality, an exception can also be seen with the following command line in the nunit-console:

"NUnitGuineaPig.dll" "NUnitGuineaPig.dll" --inprocess

In this case, "NUnitGuineaPig.dll just contained a single class which was:

        [TestCaseSource(nameof(FilesInCurrentDirectory))]
        public void FileExists(string name)  { }

        public static IEnumerable FilesInCurrentDirectory
        {
            get { yield return TestContext.CurrentContext.TestDirectory; }
        }

Running --inprocess instead gives the below exception - I imagine it's likely the two are related however:

1) Invalid : NUnitGuineaPig.TestClass.FileExists
System.AppDomainUnloadedException : The target application domain has been unloaded.

[ChrisMaddock]

@yaakov-h I believe this is the issue, however, I'm not sure what the solution should be!

nunit/nunit#2614

[ChrisMaddock]

I'm closing this in favour of nunit/nunit#2614 - now @CharliePoole's confirmed my suspicions. @yaakov-h - please feel free to follow the new issue, and if it doesn't fix your problem once it's in, we can revisit this issue.