Using Debug.Assert may crash Xunit with a mangled exception message

[ericwj]

Right-click Run test from Test Explorer may crash Xunit with a mangled exception message, do nothing, or occassionally succeed if the test contains a call to Debug.Assert that fails.

using System.Diagnostics;
using Xunit;

namespace XUnitTestProject
{
    public class XunitCrash {
        [Fact]
        public void DebugAssertFailure() => Debug.Assert(false);
    }
}

The Ouput window contains the following output

[03-Oct-17 23:15:09 Informational] ========== Discover test finished: 86 found (0:00:01.5975771) ==========
[03-Oct-17 23:15:14 Informational] ------ Run test started ------
[03-Oct-17 23:15:15 Informational] [xUnit.net 00:00:00.4643893]   Starting:    XUnitTestProject
[03-Oct-17 23:15:21 Error] The active test run was aborted. Reason: .3.0.3809, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestMethodsAsync>d__38<System.__Canon> ByRef)
   at Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTestMethodsAsync()
   at Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestClassRunner`1+<RunAsync>d__37[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.3.0.3809, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__37<System.__Canon> ByRef)
   at Xunit.Sdk.TestClassRunner`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunTestClassesAsync>d__28[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.3.0.3809, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunTestClassesAsync>d__28<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunTestClassesAsync()
   at Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Xunit.Sdk.TestCollectionRunner`1+<RunAsync>d__27[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], xunit.execution.dotnet, Version=2.3.0.3809, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c]](<RunAsync>d__27<System.__Canon> ByRef)
   at Xunit.Sdk.TestCollectionRunner`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].RunAsync()
   at Xunit.Sdk.XunitTestAssemblyRunner+<>c__DisplayClass14_2.<RunTestCollectionsAsync>b__2()
   at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.ExecuteEntry()
   at System.Threading.Tasks.SynchronizationContextTaskScheduler+<>c.<.cctor>b__8_0(System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.RunOnSyncContext(System.Threading.SendOrPostCallback, System.Object)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at Xunit.Sdk.MaxConcurrencySyncContext.WorkerThreadProc()
   at Xunit.Sdk.XunitWorkerThread+<>c.<QueueUserWorkItem>b__5_0(System.Object)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

[03-Oct-17 23:15:21 Informational] ========== Run test finished: 0 run (0:00:06.9766678) ==========

[ericwj]

It'll break when run with right-click Debug Test and succeed after continuing from there and fail as described with right-click Run from Test Explorer or when run with dotnet test.

[bradwilson]

The Debug.Assert failure is causing a process crash, which is correctly reported. Clearly this is not a normal test failure, so I'm reluctant to see why we would add a special case here.

[ericwj]

Well yes but where is the reason string coming from? Its consistently wrong and it wasn't clear to me why this happened.

If I would've known it was due to the Debug.Assert failing, I would've given priority to fixing that issue in my own code instead of fighting with test explorer and narrowing this down and writing test cases for Unsafe.AsRef and so forth. I assumed there was a memory corruption, after all. I think there still is one. It just has nothing to do with my code.

The differences in the reason message, by the way, is due to using 3 different versions of Xunit. I've had the one from June, the beta5 and just today or yesterday the rc1.

[bradwilson]

I don't know where the reason string is coming from. "The active test run was aborted" is a message that comes from VSTest, not xUnit.net.

[ericwj]

Mmm, so this is an issue with VSTest. If I run this test with dotnet test on .NET 4.6 it gives me a dialog box and the test succeeds if I click Ignore. But still running it from VS it will crash as described. Same for a .NET Framework test project indeed.

[ericwj]

Using Debug.Assert may crash test run with a mangled exception message