dotnet test swallows xUnit tests

[DerAlbertCom]

dotnet swallows silently xunit tests (maybe also test from other test frameworks) when there is a problem on the test discovery. I filed the same thing in the dotnet cli repo: https://github.com/dotnet/cli/issues/12187

Steps to reproduce

Checkout the repository https://github.com/DerAlbertCom/MisbehavingUnittestRunners run dotnet test. More details why the problem occurs is also there.

Expected behavior

At least the exception within the test discovery shows up. It would be better if it is possible to give more detail on exact type or test method.

Actual behavior

In the sample repository are 6 Unit Test, 2 of them get swallowed without any hint because of an exception within the test discovery on one test.

Environment data

.NET Core SDK (reflecting any global.json):                                                                              Version:   2.2.301
 Commit:    70d6be0814

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.301\

Host (useful for support):
  Version: 3.0.0-preview7-27912-14
  Commit:  4da6ee6450

.NET Core SDKs installed:
  2.1.508 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.108 [C:\Program Files\dotnet\sdk]
  2.2.204 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.0.100-preview7-012821 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview7.19365.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview7-27912-14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]    
To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[DerAlbertCom]

In microsoft/vstest#2123 is mentioned that the test adapter seems to capture the exception during discovery, because of that the Test2 and Test3 are not run and lost.

[DerAlbertCom]

I took a look at the SourceCode of the VsTestRunner.

The Exception Handling while

visualstudio.xunit/src/xunit.runner.visualstudio/VsTestRunner.cs

Line 247 in 8b6ddc9

catch (Exception e)

and

visualstudio.xunit/src/xunit.runner.visualstudio/VsTestRunner.cs

Line 316 in 8b6ddc9

catch (Exception e)

An Exception should be logged.

But on the diag logs of dotnet test the doesn't appear.

Theses are the only logs that the VsTestRunners seems to log to the testrunner from VS and dotnet test.

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1 (64-bit .NET Core 4.6.27817.03)
[xUnit.net 00:00:00.33]   Discovering: XUnitTestProject1
[xUnit.net 00:00:00.38]   Discovered:  XUnitTestProject1
[xUnit.net 00:00:00.39]   Starting:    XUnitTestProject1
[xUnit.net 00:00:00.46]     XUnitTestProject1.UnitTest2.Test10 [SKIP]
[xUnit.net 00:00:00.46]       Skipped with Fact
[xUnit.net 00:00:00.47]     XUnitTestProject1.UnitTest1.Test1 [SKIP]
[xUnit.net 00:00:00.47]       Skipped with Fact
[xUnit.net 00:00:00.49]     XUnitTestProject1.UnitTest2.Test20 [SKIP]
[xUnit.net 00:00:00.49]       Skipped
[xUnit.net 00:00:00.49]   Finished:    XUnitTestProject1

So, the Exception or whatever the reason is for not running Test2 and Test3 will be swallowed, a level deeper.

[DerAlbertCom]

moved to https://github.com/xunit/xunit/issues/1998