Skip to content

Audit check: an MTP-based dotnet test project with no global.json runner opt-in #1108

Description

@ptr727

Deferred from #1088. The hub fix (PR #1107) migrates the reusable validator's unit-test step to native MTP and states the consumer-side prerequisites in WORKFLOW.md D1.6 and the csharp.coverage.codecov assert, but those prerequisites are currently verified by hand only.

What Should Be Detected

A dotnet repo whose test project is MTP-based (Microsoft.Testing.Platform.MSBuild in the graph, directly or transitively, in practice xunit.v3 4.0.0 and later) but which carries no root global.json test.runner opt-in. That is the exact drift #1088 exists to prevent finding out about only when a dependabot bump PR goes red, which is how Utilities, PlexCleaner, and NxWitness each found out.

Worth detecting alongside it, since they are the same migration and the same read:

  • xunit.runner.visualstudio still referenced by an MTP-based test project (the VSTest adapter MTP replaces).
  • coverlet.collector still referenced by an MTP-based test project. Its VSTest data collector is silently ignored under MTP (MTP0001), so coverage goes dark without failing.
  • Microsoft.Testing.Extensions.CodeCoverage below the 18.9.0 floor D1.6 states. Below 18.1.0 the failure is loud (a TypeLoadException, zero tests, non-zero exit), but it still writes a well-formed Cobertura file claiming full coverage, so anything keying on the report's presence rather than the exit code reads it as a clean run.

Why It Is Not Trivial

Per the existing mechanics, spec/audit.py currently evaluates no check belonging to a type in spec/project-types.json, and AUDIT.md says so explicitly in two places. Mechanizing a csharp.* check contradicts that standing claim, so this needs AUDIT.md and the two matching statements in spec/audit.py reconciled as part of the same change, not just a new block in audit_repo.

There is also no existing "does this repo have a test project" signal in the registry or the spec. The nearest proxy is the coverage_active derivation, which really means "declares csharp or python at build profile". Detecting a test project means scanning the tree via repo_tree_entries for *Tests*.csproj, which the file-presence dimension already fetches once and can be reused rather than re-fetched.

Notes

  • The check has parseable logic, so per the fleet's own convention it should be a module-level helper returning (kind, text) tuples with a case table in _selftest, rather than an inline block in audit_repo (neither the dependabot nor the cspell inline blocks have self-test coverage today).
  • Verdict tier is a judgment call worth making deliberately: a repo not yet migrated is a migration owed rather than drift, per the wording PR Migrate the Hub's Unit-Test Step to Native Microsoft.Testing.Platform #1107 lands in D1.6, so DRIFT is probably right and LETTER probably is not, at least until the fleet rollout completes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    gateA rule with no mechanical check, or a check that misses a shape

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions