dotnet test and VsTest swallows xUnit tests

[DerAlbertCom]

copied from xunit/visualstudio.xunit#164 and xunit/visualstudio.xunit#165.

xunit swallows silently xunit tests (maybe also test from other test frameworks) when there is a problem on the test discovery. I filed the same thing in the dotnet cli repo: https://github.com/dotnet/cli/issues/12187

Steps to reproduce

Checkout the repository https://github.com/DerAlbertCom/MisbehavingUnittestRunners run dotnet test. More details why the problem occurs is also there.

Expected behavior

At least the exception within the test discovery shows up. It would be better if it is possible to give more detail on exact type or test method.

Actual behavior of dotnet test

In the sample repository are 6 Unit Test, 2 of them get swallowed without any hint because of an exception within the test discovery on one test.

Actual behavior of Visual Studio

Test withing Visual Studio 2019 16.2
.
In the sample repository are 6 Unit Test, 2 of are not run without any hint because of an exception within the test discovery on one test.

Environment data

.NET Core SDK (reflecting any global.json):                                                                              Version:   2.2.301
 Commit:    70d6be0814

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.301\

Host (useful for support):
  Version: 3.0.0-preview7-27912-14
  Commit:  4da6ee6450

.NET Core SDKs installed:
  2.1.508 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.108 [C:\Program Files\dotnet\sdk]
  2.2.204 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100-preview7-012821 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview7.19365.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]    
To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[DerAlbertCom]

In microsoft/vstest#2123 is mentioned that the test adapter seems to capture the exception during discovery, because of that the Test2 and Test3 are not run and lost.

I took a look at the SourceCode of the VsTestRunner.

The Exception Handling while
https://github.com/xunit/visualstudio.xunit/blob/8b6ddc907a600dcad3e6f961486e54d60283e86d/src/xunit.runner.visualstudio/VsTestRunner.cs#L247

and

https://github.com/xunit/visualstudio.xunit/blob/8b6ddc907a600dcad3e6f961486e54d60283e86d/src/xunit.runner.visualstudio/VsTestRunner.cs#L316

An Exception should be logged.

But on the diag logs of dotnet test the doesn't appear.

Theses are the only logs that the VsTestRunners seems to log to the testrunner from VS and dotnet test.

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 4.6.27817.03)
[xUnit.net 00:00:00.33]   Discovering: XUnitTestProject1
[xUnit.net 00:00:00.38]   Discovered:  XUnitTestProject1
[xUnit.net 00:00:00.39]   Starting:    XUnitTestProject1
[xUnit.net 00:00:00.46]     XUnitTestProject1.UnitTest2.Test10 [SKIP]
[xUnit.net 00:00:00.46]       Skipped with Fact
[xUnit.net 00:00:00.47]     XUnitTestProject1.UnitTest1.Test1 [SKIP]
[xUnit.net 00:00:00.47]       Skipped with Fact
[xUnit.net 00:00:00.49]     XUnitTestProject1.UnitTest2.Test20 [SKIP]
[xUnit.net 00:00:00.49]       Skipped
[xUnit.net 00:00:00.49]   Finished:    XUnitTestProject1

So, the Exception or whatever the reason is for not running Test2 and Test3 will be swallowed a level deeper.

[DerAlbertCom]

Any Ideas, About this?

[DerAlbertCom]

ReSharper relies also on the results of xunit and behaves not well because of that, https://youtrack.jetbrains.com/issue/RSRP-475867

[bradwilson]

I should not have transferred this, as it appears to be reproducible without VSTest, sooooo... transferring it back to the core project.

[aomader]

@bradwilson Is this issue really fixed? We noticed a similar behavior, i.e., swallowed tests caused by an exception outside of the actual test code.

A minimal example that reproduces the issue:

public class CustomFactAttribute : FactAttribute
{
    public override string? Skip => throw new InvalidOperationException("I would like to see this issue...");
}

public class UnitTest1
{
    [CustomFact]
    public void Test1()
    {
        Assert.Fail("This test should fail.");
    }
}

If run via ReSharper, we can see the exception and the inconclusive test:

If instead run via dotnet test, the test is silently swallowed:

$ dotnet test
  Determining projects to restore...
  All projects are up-to-date for restore.
  xunitbug -> /Users/MAO1LB/Downloads/xunitbug/bin/Debug/net8.0/xunitbug.dll
Test run for /Users/MAO1LB/Downloads/xunitbug/bin/Debug/net8.0/xunitbug.dll (.NETCoreApp,Version=v8.0)
Microsoft (R) Test Execution Command Line Tool Version 17.8.0 (arm64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in /Users/MAO1LB/Downloads/xunitbug/bin/Debug/net8.0/xunitbug.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

[bradwilson]

This must be a different issue, I'll see what I can track down.

[bradwilson]

Fixed in v2 2.8.1-pre.8 https://xunit.net/docs/using-ci-builds

I have other work to do with v3 related to this that will probably take a bit longer, as it's failing in a different way, so I'm going to leave the bug open for now until I can resolved the v3 issue(s).

@aomader Your example now:

[bradwilson]

Fixed in v3 0.1.1-pre.420.

[aomader]

@bradwilson Awesome, thank you very much! Probably one of the speediest replies PLUS fix I have ever seen on GitHib. Kudos! 🥇