Public API for coverage extensions: threshold checks, terminal summary, and exit code in Microsoft.Testing.Platform

[james-newell-forge]

Summary

Add a public API in Microsoft.Testing.Platform (MTP) so coverage extensions can perform coverage-threshold checks and surface coverage results in the terminal summary and threshold checks in the process exit code. Adds two message-bus types, three enums, and a new exit code.

Proposed in PR #9896 — filing here so the team can approve the public API shape before it ships (MTP public API is effectively permanent).

Background and Motivation

Today a coverage extension cannot:

• Enforce a coverage threshold — there's no platform contract to report "coverage X% is below the required Y%", and no way to fail the run for it.
• Show coverage in the standard terminal summary — the built-in terminal reporter has no coverage concept, so a coverage tool must print its own ad-hoc output.
• Influence the process exit code — there's no dedicated, ignorable exit code for a threshold failure, so each tool invents its own convention.

Coverage is produced out-of-process: a coverage extension registers as an ITestHostProcessLifetimeHandler (runs in the test-host controller) and publishes these messages after the test host exits; the controller then renders the summary and applies the verdict. A stable public message contract is what lets an external coverage extension plug into that pipeline.

Example coverage extensions that would benefit from this change:

• https://github.com/microsoft/codecoverage
• https://github.com/coverlet-coverage/coverlet

Proposed Feature

New public API in Microsoft.Testing.Platform.Extensions.Messages:

public enum CoverageType { Line = 0, Branch = 1, Method = 2 }
public enum CoverageThresholdStatus { Passed = 0, Failed = 1 }
public enum CoverageThresholdStatistic { Minimum = 0, Total = 1, Average = 2 }

public sealed class TestCoverageMessage : PropertyBagData
{
    public TestCoverageMessage(string moduleName, double value, CoverageType coverageType);
    public string ModuleName { get; }
    public double Value { get; }          // percentage, 0..100
    public CoverageType CoverageType { get; }
}

public sealed class TestCoverageThresholdMessage : PropertyBagData
{
    public TestCoverageThresholdMessage(double value, double threshold,
        CoverageType coverageType, CoverageThresholdStatus status,
        CoverageThresholdStatistic statistic);
    public double Value { get; }
    public double Threshold { get; }
    public CoverageType CoverageType { get; }
    public CoverageThresholdStatus Status { get; }
    public CoverageThresholdStatistic Statistic { get; }
}

Plus a new exit code Microsoft.Testing.Platform.Helpers.ExitCode.CoverageThresholdFailed = 14.

Behavior: coverage extensions publish these messages; the terminal device renders a coverage summary + threshold results, and a failed threshold on an otherwise-successful run sets exit code 14, routed through the existing --ignore-exit-code policy so --ignore-exit-code 14 suppresses it.

Alternative Designs

• No platform API (coverage extension owns everything). Rejected — a tool can't integrate with the built-in terminal reporter or --ignore-exit-code, and every tool diverges on output and exit codes.
• In-process coverage production. Rejected — collectors run in the controller around the test-host process, so the out-of-process model is the realistic one.
• Reuse a generic/combined message. Rejected — coverage results and threshold verdicts are distinct, independently produced concepts that benefit from strongly-typed contracts.

[Evangelink]

Thanks for opening the tracking issue @james-newell-forge!

Picking up the API-shape discussion from #9896 — before we lock in a permanent public surface, I'd like to suggest reconsidering the message types, in line with the review feedback there:

• Base type. TestCoverageMessage and TestCoverageThresholdMessage both derive from PropertyBagData but never populate the bag, so a consumer inspecting the bag generically sees nothing. I'd suggest we either carry the coverage data as actual bag properties, or move these to a base type that better fits a strongly-typed contract instead of an always-empty PropertyBagData. My preference leans toward a dedicated base for clarity.
• Naming. Since the API is effectively permanent, let's make sure the members read well (e.g. Statistic rather than any abbreviation, and unambiguous enum names). The issue text already uses Statistic/CoverageThresholdStatistic, which is the direction I'd want.

To be clear, this is still open for discussion — I'd like the team (and you) to weigh in on the base-type choice in particular before more implementation effort goes in. What are your thoughts?