Fail AzDO Pipeline if number of passed tests fall below a certain threashold.

[halter73]

Is your feature request related to a problem? Please describe.

This week, the number of passed tests as part of our daily CI builds and PR checks this week decreased from ~85,000 to ~20,000. Despite this, the PR checks passed an the build stayed green.

The tests stopped running because of an issue with xunit/arcade (see dotnet/arcade#5908) that caused dotnet test runs to fail silently, but I think automated checks should have caught the dramatic reduction in the number of passed tests.

Describe the solution you'd like

We should set some sort of minimum threshold on the number of passed tests for a CI build/PR check pass. If we ever intentionally reduce the number of tests we run intentionally, we can manually adjust this threshold.

[dougbu]

@ilyas1974 @riarenas @chcosta is this something that the Arcade SDK could provide centrally❔ I'd rather we didn't build something specific in dotnet/aspnetcore for what seems like a very general and centralizable (sic) request.

[ilyas1974]

@markwilkie, is this something that is part of the resiliency epic (dotnet/arcade#5963) that we are starting to work on?

[chcosta]

What's the acceptable number of tests that can fail silently? I agree that something could be done here, I'm just not sure what that solution is, or in this case; how you'd draw the line given a static number. Seems like this would just shift the problem slightly.

That's just my 2 cents. I do think that this problem space is worth considering further.

[dougbu]

What's the acceptable number of tests that can fail silently?

I'd emphasize this is about broad infrastructure issues e.g. bugs in test runners and not test failures per se. If results aren't reported at all, we need a fallback to ensure the pipeline fails.

But, could Arcade expose a setting allowing the minimum number of expected tests to be set❔ For example, our quarantined run may run just one test for a test assembly containing no quarantined tests. (We have always-successful tests for the quarantined and not categories to ensure the results files are not empty unless we hit another silent failure case.)

[dougbu]

This may be a good idea but isn't something we should handle on a per-repo basis. Suggest moving this issue to dotnet/core-eng or dotnet/arcade. @markwilkie should this be something on your radar e.g. part of the M1 telemetry you're working on❔

If I don't hear back, I'll move this to Core-Eng tomorrow.