Runtime.Remoting.RemotingException in NUnit.Engine.Runners.ProcessRunner.Dispose

[lsem]

Hi,

I'm trying to migrate my windows build server to linux and now have crashes in my unit tests:

khadmin@khbuild:~/myagent/_work/2/s$ mono  ./thirdparty/tools/NUnit/nunit3-console.exe --labels=All --noheader ./buildartifacts/KH.Tests.dll /result=./buildartifacts/EventsCursorTestsResults.xml
Runtime Environment
   OS Version: Linux 4.4.0.62
  CLR Version: 4.0.30319.42000

Test Files
    ./buildartifacts/KH.Tests.dll

=> KH..Tests.when_something.test_xx
System.Runtime.Remoting.RemotingException: Tcp transport error. ---> System.Runtime.Remoting.RemotingException: Connection closed
  at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.StreamRead (System.IO.Stream networkStream, System.Byte[] buffer, System.Int32 count) [0x00014] in <d849815271d74866b18b5a29e4184d0c>:0
  at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.ReceiveMessageStatus (System.IO.Stream networkStream, System.Byte[] buffer) [0x00000] in <d849815271d74866b18b5a29e4184d0c>:0
   --- End of inner exception stack trace ---


Server stack trace:
  at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.ReceiveMessageStatus (System.IO.Stream networkStream, System.Byte[] buffer) [0x0001a] in <d849815271d74866b18b5a29e4184d0c>:0
  at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage (System.Runtime.Remoting.Messaging.IMessage msg, System.Runtime.Remoting.Channels.ITransportHeaders requestHeaders, System.IO.Stream requestStream, System.Runtime.Remoting.Channels.ITransportHeaders& responseHeaders, System.IO.Stream& responseStream) [0x00061] in <d849815271d74866b18b5a29e4184d0c>:0
  at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage (System.Runtime.Remoting.Messaging.IMessage msg) [0x0006c] in <d849815271d74866b18b5a29e4184d0c>:0

Exception rethrown at [0]:
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_remoting_wrapper (intptr,intptr)
  at (wrapper remoting-invoke) NUnit.Engine.Agents.RemoteTestAgent:Stop ()
  at NUnit.Engine.Runners.ProcessRunner.Dispose (System.Boolean disposing) [0x0004b] in <7b0953727751470ab10f0e0b547b85ae>:0

May be this is known issue?

[lsem]

Sorry, I'm closing this until make sure I have proper version. Never mind.

[lsem]

So, I've checked version:

khadmin@khbuild:~/myagent/_work/2/s$ mono thirdparty/tools/NUnit/nunit3-console.exe --version
NUnit Console Runner 3.6.1
Copyright (C) 2017 Charlie Poole 

OS and Mono:

khadmin@khbuild:~/myagent/_work/2/s$ uname -a
Linux khbuild 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
khadmin@khbuild:~/myagent/_work/2/s$ mono --version
Mono JIT compiler version 4.8.0 (Stable 4.8.0.524/9d74414 Mon Apr 10 07:40:19 UTC 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug
	LLVM:          yes(3.6.0svn-mono-/)
	GC:            sgen

[tom-dudley]

This looks to be very much related to issue nunit/nunit#1834 where we get a similar stack trace about a closed connection when disposing. Obviously this is slightly different since you're running mono and not using the explore flag, but I'd hazard a guess that they're the same underlying bug.

[rprouse]

I put a confirm label on this because I think it is likely a duplicate of nunit/nunit#1834

[lsem]

I've spent some time and now have more information about cause of this issue.

There is a race between ProcessRunner.Dispose(bool) which calls at some point something like:

try { 
   _agent.Stop();       // This fires up manual-reset-event
   _agent = null;        // This effectively disposes something related to .net-remoting-tcp subsystem 
                                  // which probably wants to gracefully close tcp connection  
} catch  (Exception e) {
    if ( unloadError == null) { throw; } // This is exception we see in my report. 
}

And nunit-agents/Program.cs with code like:

...
                try
                {
                    if (Agent.Start())
                        WaitForStop();
                    else
                        log.Error("Failed to start RemoteTestAgent");
                }
                catch (Exception ex)
                {
                    log.Error("Exception in RemoteTestAgent", ex);
                }

                //log.Info("Unregistering Channel");
                try
                {
                    ChannelServices.UnregisterChannel(Channel);
                }
                catch (Exception ex)
                {
                    log.Error("ChannelServices.UnregisterChannel threw an exception", ex);
                }

So, my guess is that problem is that remote runner process receives stop signal and immiditely closes tcp connection while Host process tries gracefully finalize tcp connection which is already closed at that moment.

Just to work around, I can either ignore that exception in ProcessRunner.Dispose() or put pause in nunit-agent/Program.cs and both should help.

Sure, I could be wrong with all that, but quick investigation ended up with such conclusion.

[jnm2]

Looks like a lead! I'm surprised the _agent = null; line does anything to the remoting system.

[tom-dudley]

I don't think the _agent = null line is relevant here. _agent.Stop() does the disposing (and the throwing) so we don't actually get to executing the next line.

[lsem]

Yes, _agent = null is indeed unrelated and hypothesis turned out to be wrong.
What is still true is race between Dispatch and Remote Process closing channel.

I don't know how does signaling stop event in host process and waiting in child process work, but can assume it might have underlying tcp communication involved via the channel that is closed by child process just after receiving signal. So that, Host (from dispose) sends Stop() this stop is not atomic, for example it sends asynchronous signal, remote side receives it unblocks from wait and closes channel while host reads something from that channel to complete Send().

This is just the only hypothesis I have for now, I don't know how this Remoting stuff works internally and even not sure I correctly followed relations between different nunit models (not familiar with codebase). So, excuse me if I'm wrong and waste your time.

And this hypothesis is likely wrong but what is probably correct that signaling stop event on host side is not atomic, it makes remote side woken up while its further activity (whether it is closing channel or just exiting) makes host Send throwing exception.

[jnm2]

@lsem I believe you are right about the core of the problem. While the console is in the process of making the remoting call to RemoteTestAgent.Stop, all RemoteTestAgent.Stop does is set a signal and the main thread which had been waiting on the signal immediately starts tearing things down.

I have just proven this. All you need is to put a delay in between the two to make sure remoting has returned, and the error vanishes.

[jnm2]

Next up: familiarize myself with remoting best practices around shutdown.

Same question, no answer: https://social.msdn.microsoft.com/Forums/en-US/e8fd1a5c-ee9d-4e37-9773-23193dfa30a0

There does not seem to be any way to synchronize with the remoting logic to make sure that you don't shut down while trying to return information.

If there's a way to decorate IServerChannelSink, we could snoop to find out when the TCP response is finished being sent. It seems overly complex but without that, we have to choose between a very safe delay or swallowing the SocketException. @CharliePoole @rprouse @ChrisMaddock thoughts?

I stepped through in the debugger. We basically need to shut down no sooner than this line finishing:
http://referencesource.microsoft.com/#System.Runtime.Remoting/channels/tcp/tcpserverchannel.cs,613

Perhaps writing an IServerChannelSink is not as extraordinary as I thought for people dealing with remoting.
https://stackoverflow.com/questions/527676/#comment2066979_527756

[jnm2]

I'm almost there but I have to call it for the night. The only hitch is that there are two channels being created and NUnitTestAgent.Channel doesn't seem to be used at all. I'm going to have to add the counter to ServiceBase.

[ChrisMaddock]

Fingers crossed @jnm2 - sounds like a sensible solution. Cheers for looking in to this one, it'll be great to no longer see this error!

[jnm2]

#223 is ready for review!

[lsem]

@jnm2 I can confirm that putting delay fixes (works around) this particular problem. That was what I actually did for my build server.
Thanks for better fix!

[jnm2]

We believe this was fixed by #223 which will be released as part of 3.7. If you'd like to confirm sooner that the issue is fixed, please try 3.7.0-dev-03641 from the https://ci.appveyor.com/nuget/nunit-console NuGet feed.

If it turns out that this does not fix your issue, please reopen and comment.

[craigfowler]

I am still seeing this in NUnit.ConsoleRunner v3.7.0 (from NuGet) using Linux/Mono. It's intermittent and doesn't crash every time.

NUnit Console Runner 3.7.0 
Copyright (c) 2017 Charlie Poole, Rob Prouse
Runtime Environment
   OS Version: Linux 4.11.6.41106 
  CLR Version: 4.0.30319.42000
Test Files
    Tests/CSF.Screenplay.Web.Tests/bin/Debug/CSF.Screenplay.Web.Tests.dll
System.Runtime.Remoting.RemotingException: Tcp transport error. ---> System.Runtime.Remoting.RemotingException: Connection closed
  at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.StreamRead (System.IO.Stream networkStream, System.Byte[] buffer, System.Int32 count) [0x00011] in <ffd0b3fb04044b19b8235de76cf332b2>:0 
  at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.ReceiveMessageStatus (System.IO.Stream networkStream, System.Byte[] buffer) [0x00000] in <ffd0b3fb04044b19b8235de76cf332b2>:0 
   --- End of inner exception stack trace ---
Server stack trace: 
  at System.Runtime.Remoting.Channels.Tcp.TcpMessageIO.ReceiveMessageStatus (System.IO.Stream networkStream, System.Byte[] buffer) [0x00017] in <ffd0b3fb04044b19b8235de76cf332b2>:0 
  at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage (System.Runtime.Remoting.Messaging.IMessage msg, System.Runtime.Remoting.Channels.ITransportHeaders requestHeaders, System.IO.Stream requestStream, System.Runtime.Remoting.Channels.ITransportHeaders& responseHeaders, System.IO.Stream& responseStream) [0x0005e] in <ffd0b3fb04044b19b8235de76cf332b2>:0 
  at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage (System.Runtime.Remoting.Messaging.IMessage msg) [0x00066] in <ffd0b3fb04044b19b8235de76cf332b2>:0 
Exception rethrown at [0]: 
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_remoting_wrapper (intptr,intptr)
  at (wrapper remoting-invoke) NUnit.Engine.Agents.RemoteTestAgent:Stop ()
  at NUnit.Engine.Runners.ProcessRunner.Dispose (System.Boolean disposing) [0x00086] in <1e8ad6af4c6f4686ad7e5f9e67020b3b>:0 
  at NUnit.Engine.Runners.AbstractTestRunner.Dispose () [0x00000] in <1e8ad6af4c6f4686ad7e5f9e67020b3b>:0 
  at NUnit.Engine.Runners.MasterTestRunner.Dispose (System.Boolean disposing) [0x00013] in <1e8ad6af4c6f4686ad7e5f9e67020b3b>:0 
  at NUnit.Engine.Runners.MasterTestRunner.Dispose () [0x00000] in <1e8ad6af4c6f4686ad7e5f9e67020b3b>:0 
  at NUnit.ConsoleRunner.ConsoleRunner.RunTests (NUnit.Engine.TestPackage package, NUnit.Engine.TestFilter filter) [0x0010e] in <5d13e9f4d03e4da1b4779cb1d61b9b3d>:0 
  at NUnit.ConsoleRunner.ConsoleRunner.Execute () [0x000b6] in <5d13e9f4d03e4da1b4779cb1d61b9b3d>:0 
  at NUnit.ConsoleRunner.Program.Main (System.String[] args) [0x001bf] in <5d13e9f4d03e4da1b4779cb1d61b9b3d>:0 

I have a log of the whole build used to reproduce this available on Travis. In that log, the crash above is recorded starting on line 2784. I think that the rest of the log should contain just about everything else about the build environment you might possibly want to know.

[jnm2]

@craigfowler Thank you for that. You're not the only one now. Please see #255 (comment) - if you're able, testing the runner in that comment with --remoting-wait-after-stop --remoting-shutdown-delay=5000 would give us useful information and might work around your immediate problem.

[jnm2]

@rprouse Or should I suggest he try the runner from https://travis-ci.org/nunit/nunit-console/builds/266292088?

[rprouse]

@jnm2 I don't think my changes will help for @craigfowler. You cannot install on Linux, so it won't find another engine and my repro happens all the time.

[craigfowler]

OK, I'm set up using that build of 3.8.0 mentioned in the comment of #255 and all of my Linux/Travis builds will use that test runner for now.

So far so good (that is - I've seen at least one success), although of course it being intermittent I'll have to wait a while with an eye out for failures.

One thing - and tbh I'm not sure if it's NUnit messaging or Selenium WebDriver (I don't know either's messages well enough to tell them apart). In lines 2790 and 2791 of this build I get the following warnings, although the build DID pass without error:

Unable to unload AppDomain, Unload thread timed out.
Agent Process was terminated successfully after error.

That might be a red herring though, if it's me being dumb and doing something wrong with a webdriver, or perhaps it is related to this issue?

It's worth saying that I have a fair bit going on in an ITestAction.AfterTest(ITest) method which is applied via attribute at the assembly level (perhaps this is happening at the same time as unload). If so, could this be contributing to busting timeouts and exacerbating race conditions?

Not familiar enough with NUnit's architecture to know if that's useful information or just tangential waffle.

[jnm2]

Great news, keep us posted!

Unable to unload AppDomain, Unload thread timed out.
Agent Process was terminated successfully after error.

I didn't see this when I ran that build, but the temporary fix is --remoting-shutdown-delay=5000 which means the agent process sleeps for 5000 ms to avoid shutting down the socket too soon since there is literally no other avenue to avoiding the SocketException that I can find without getting rid of remoting entirely, which I'm in the process of working on. The point is, you may be able to make that number smaller and still keep the build succeeding, perhaps even --remoting-shutdown-delay=100, and that might avoid the above messages. I don't know because I can't repro.

[craigfowler]

Update: I've just seen a failure using the experimental console runner build linked above. It's logged at this Travis build, starting on line 2678.

This is using the recommended command-line switches as well. The actual command-line used to run these tests is available in the build script line 34.

NUnit Console Runner 3.8.0 
Copyright (c) 2017 Charlie Poole, Rob Prouse
Runtime Environment
   OS Version: Linux 4.11.6.41106 
  CLR Version: 4.0.30319.42000
Test Files
    ./Tests/CSF.Screenplay.Reporting.Tests/bin/Debug/CSF.Screenplay.Reporting.Tests.dll
    ./Tests/CSF.Screenplay.SpecFlow.Tests/bin/Debug/CSF.Screenplay.SpecFlow.Tests.dll
    ./Tests/CSF.Screenplay.Tests/bin/Debug/CSF.Screenplay.Tests.dll
    ./Tests/CSF.WebDriverFactory.Tests/bin/Debug/CSF.WebDriverFactory.Tests.dll
Errors, Failures and Warnings
1) Error : /home/travis/build/csf-dev/CSF.Screenplay/Tests/CSF.Screenplay.Reporting.Tests/bin/Debug/CSF.Screenplay.Reporting.Tests.dll
The object with ID 2 implements the IObjectReference interface for which all dependencies cannot be resolved. The likely cause is two instances of IObjectReference that have a mutual dependency on each other.
Server stack trace: 
  at System.Runtime.Serialization.ObjectManager.GetCompletionInfo (System.Runtime.Serialization.FixupHolder fixup, System.Runtime.Serialization.ObjectHolder& holder, System.Object& member, System.Boolean bThrowIfMissing) [0x000ae] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.ObjectManager.CompleteObject (System.Runtime.Serialization.ObjectHolder holder, System.Boolean bObjectFullyComplete) [0x00134] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.ObjectManager.DoFixups () [0x000f1] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize (System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Runtime.Serialization.Formatters.Binary.__BinaryParser serParser, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x00077] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x000a2] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x00000] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck) [0x00000] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) [0x00000] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage (System.Runtime.Remoting.Channels.IServerChannelSinkStack sinkStack, System.Runtime.Remoting.Messaging.IMessage requestMsg, System.Runtime.Remoting.Channels.ITransportHeaders requestHeaders, System.IO.Stream requestStream, System.Runtime.Remoting.Messaging.IMessage& responseMsg, System.Runtime.Remoting.Channels.ITransportHeaders& responseHeaders, System.IO.Stream& responseStream) [0x0009d] in <ffd0b3fb04044b19b8235de76cf332b2>:0 
Exception rethrown at [0]: 
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_remoting_wrapper (intptr,intptr)
  at (wrapper remoting-invoke) NUnit.Engine.Agents.RemoteTestAgent:Run (NUnit.Engine.ITestEventListener,NUnit.Engine.TestFilter)
  at NUnit.Engine.Runners.ProcessRunner.RunTests (NUnit.Engine.ITestEventListener listener, NUnit.Engine.TestFilter filter) [0x00025] in <18c4d2198635420998bb9b3c03b77836>:0 
2) Error : /home/travis/build/csf-dev/CSF.Screenplay/Tests/CSF.Screenplay.SpecFlow.Tests/bin/Debug/CSF.Screenplay.SpecFlow.Tests.dll
The object with ID 2 implements the IObjectReference interface for which all dependencies cannot be resolved. The likely cause is two instances of IObjectReference that have a mutual dependency on each other.
Server stack trace: 
  at System.Runtime.Serialization.ObjectManager.GetCompletionInfo (System.Runtime.Serialization.FixupHolder fixup, System.Runtime.Serialization.ObjectHolder& holder, System.Object& member, System.Boolean bThrowIfMissing) [0x000ae] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.ObjectManager.CompleteObject (System.Runtime.Serialization.ObjectHolder holder, System.Boolean bObjectFullyComplete) [0x00134] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.ObjectManager.DoFixups () [0x000f1] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize (System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Runtime.Serialization.Formatters.Binary.__BinaryParser serParser, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x00077] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Boolean isCrossAppDomain, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x000a2] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck, System.Runtime.Remoting.Messaging.IMethodCallMessage methodCallMessage) [0x00000] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler, System.Boolean fCheck) [0x00000] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream serializationStream, System.Runtime.Remoting.Messaging.HeaderHandler handler) [0x00000] in <204f770036d441bb8dfd3daba3550e83>:0 
  at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage (System.Runtime.Remoting.Channels.IServerChannelSinkStack sinkStack, System.Runtime.Remoting.Messaging.IMessage requestMsg, System.Runtime.Remoting.Channels.ITransportHeaders requestHeaders, System.IO.Stream requestStream, System.Runtime.Remoting.Messaging.IMessage& responseMsg, System.Runtime.Remoting.Channels.ITransportHeaders& responseHeaders, System.IO.Stream& responseStream) [0x0009d] in <ffd0b3fb04044b19b8235de76cf332b2>:0 
Exception rethrown at [0]: 
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_remoting_wrapper (intptr,intptr)
  at (wrapper remoting-invoke) NUnit.Engine.Agents.RemoteTestAgent:Run (NUnit.Engine.ITestEventListener,NUnit.Engine.TestFilter)
  at NUnit.Engine.Runners.ProcessRunner.RunTests (NUnit.Engine.ITestEventListener listener, NUnit.Engine.TestFilter filter) [0x00025] in <18c4d2198635420998bb9b3c03b77836>:0 
Test Run Summary
  Overall result: Failed
  Test Count: 30, Passed: 30, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2017-08-22 12:17:06Z
    End time: 2017-08-22 12:17:08Z
    Duration: 1.974 seconds

[CharliePoole]

In general, I send people to an appropriate dev build on MyGet. Those are already reviewed and merged, whereas appveyor builds may or may not be. The MyGet builds are like our continuous beta release toward the next version.

[srillaert]

A workaround for us is to run only one test assembly agent at a time by setting the command line option --agents=1, so :
mono ~/.nuget/packages/NUnit.ConsoleRunner/3.7.0/tools/nunit3-console.exe [inputfiles] --agents=1

This solved for us the intermittent error 'The object with ID 2 implements the IObjectReference interface for which all dependencies cannot be resolved. The likely cause is two instances of IObjectReference that have a mutual dependency on each other.'.

We are running the NUnit Console Runner 3.7.0. with mono on Ubuntu 16.04. The error only occurs intermittently when testing more than one assembly.