Debug.Fail or Debug.Assert causing entire test run to be aborted

[RobSiklos]

In our code base, we have a lot of calls to Debug.Assert() and Debug.Fail(). We also have a lot of unit tests which go through those code paths and make sure things don't break down.

When running those unit tests in .NET Framework, we could remove the DefaultTraceListener from Debug.Listeners, and the tests would run along happily.

However, in .NET Core (2.2), there doesn't seem to be any opportunity to access Debug.Listeners. When we run our unit tests, the first one that hits a Debug.Fail() or a failed Debug.Assert() will cause the rest of the entire test run to be aborted.

Steps to replicate:

1. Create a new .NET Core 2.2 project with some public methods. At least one method should call Debug.Fail().
2. Create a unit test project
3. Create two test methods. The first one calls the public method with the Debug.Fail(), the second method does nothing.
4. Run the tests

Actual Result: The first test fails to complete, and the second test doesn't execute at all. In the "Tests" category of the VS Output window, you get "The active test run was aborted. Reason: Assertion Failed" with a stack trace.

Expected Result: The first test fails to complete, but the rest of the tests keep getting run.

Note: I entered this under coreclr because I think the real issue is that we no longer have the ability to modify the debug listeners, like we could in .NET Framework.

[RussKeldorph]

@dotnet/dotnet-diag

[davmason]

In coreclr we call Environment.FailFast when an assert fires or Debug.Fail is called. You can read the discussion around why at https://github.com/dotnet/corefx/issues/3708. We are not able to provide the same experience as the desktop because we can't assume we have a UI to show, and there are also issues with how Windows Error Reporting handles failures.

We (the runtime team) don't really have any control over how the VSTest behaves, you may be able to start a discussion at the VSTest repo. I suspect they are not able to substantially change the behavior since Environment.FailFast is being called. It looks like they did some work to at least make it a diagnosable failure in the past microsoft/vstest#1688

[RobSiklos]

@davmason So what is the official guidance around unit testing a codebase which makes extensive use of Debug.Fail?

[stephentoub]

For .NET Core 3.0, Debug respects Trace.Listeners, so if you were to do Trace.Listeners.Clear(), Debug.Fail would become a nop, or if you were to add in your own listener to the collection, it would be invoked in response. @maryamariyan did that work and can provide more details as needed.

[maryamariyan]

Sample tests you can look at to see the usage:

• DebugTestsUsingListeners

• DebugTestsNoListeners

Examples:

• Example for Debug.Write and Trace.Write
• Example for Debug.Fail and Trace.Fail

As opposed to when you don't use Trace.Listeners Debug.Fail would not get skipped as shown here

Note, internally to set up the test we used WriteAssert (https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Debug/tests/DebugTests.cs#L62-L88) to mock the actual FailCore and WriteCore behavior via reflection

So as @stephentoub mentioned above, using Trace.Listeners.Clear() would make Debug.Fail nop.

cc: @jkotas

[jkotas]

This is fixed in .NET Core 3.0. Previews are available here: https://blogs.msdn.microsoft.com/dotnet/2019/01/29/announcing-net-core-3-preview-2/ . Please give it a try and let us know if you find any issues.

[RobSiklos]

Confirmed - in .NET Core 3.0, Debug respects Trace.Listeners.
Thanks all!

[RobSiklos]

@maryamariyan Thanks for the sample code. For 2.2 I'll just set the value of Debug.s_ShowDialog via reflection and we're in great shape!