Revise `--minimum-expected-tests` behavior with Retry extension

[Youssef1313]

This is a hypothesis that I didn't test yet.

But I think if --minimum-expected-tests is used with Retry, it could be problematic.

Let's say a test project has 10 tests, and --minimum-expected-tests 5 is passed. If one test failed in the initial run, the subsequent run will run only the one failed test, causing --minimum-expected-tests to potentially fail the run?

We need to revise this scenario and try to support it.

At least if it already works, we will need a test to cover it.

[Evangelink]

I would hope this is handled but thanks for the issue and let's ensure we have some acceptance test for it!

[Youssef1313]

I just tested, seems not handled.

using System.Diagnostics;

namespace TestProject87;

[TestClass]
public sealed class Test1
{
    [TestMethod]
    public void TestMethod1()
    {
    }

    [TestMethod]
    public void TestMethod2()
    {
        // --results-directory for first run will contain "Retries" in the path and ends with "1". Only fail the first retry.
        if (Environment.GetCommandLineArgs().Any(arg => arg.Contains("Retries") && arg.EndsWith("1")))
        {
            Assert.Fail("Failing");
        }
    }
}

Running with:

dotnet run -- --minimum-expected-tests 2 --retry-failed-tests 1

Output:

MSTest v3.8.3 (UTC 17/03/2025) [win-x64 - .NET 8.0.16]
failed TestMethod2 (37ms)
  Assert.Fail failed. Failing
    at TestProject87.Test1.TestMethod2() in Test1.cs:19
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 2
  failed: 1
  succeeded: 1
  skipped: 0
  duration: 274ms


=====================
Tests suite failed, total failed tests: 1, exit code: 2, attempt: 1/2
=====================


Test run summary: Minimum expected tests policy violation, tests ran 1, minimum expected 2 - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 208ms

Test suite failed with and exit code different that 2 (failed tests). Failure related to an unexpected condition. Exit code '9'

If --minimum-expected-tests not set:

MSTest v3.8.3 (UTC 17/03/2025) [win-x64 - .NET 8.0.16]
failed TestMethod2 (53ms)
  Assert.Fail failed. Failing
    at TestProject87.Test1.TestMethod2() in Test1.cs:19
    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Test run summary: Failed! - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 2
  failed: 1
  succeeded: 1
  skipped: 0
  duration: 499ms


=====================
Tests suite failed, total failed tests: 1, exit code: 2, attempt: 1/2
=====================


Test run summary: Passed! - bin\Debug\net8.0\TestProject87.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 192ms


=====================
Tests suite completed successfully in 2 attempts
=====================