Skip to content

Intent-tier checks have no mechanical floor, so a codecov.yml that gates PRs passes the audit #1110

Description

@ptr727

Raised by CodeRabbit against PR #1107 and separated out from it, since the observation is about the fidelity model rather than about that PR's MTP migration.

The Observation

A requirement stated in WORKFLOW.md and carried in spec/files.json at fidelity: intent is evaluated by a reading agent against a catalog/ reference, never mechanically. Nothing fails when the repo drifts from it. The concrete case raised was codecov.yml:

{ "path": "codecov.yml", "fidelity": "intent", "reference": "catalog/snippets/configs/codecov.yml", "intentRef": "WORKFLOW.md", "appliesTo": ["csharp", "python"] }

D1.6 requires that file to set project and patch statuses to informational: true, so a coverage delta never gates a pull request, and to exclude intentionally-untested code from the denominator via ignore. The reference snippet carries both. But a repo whose codecov.yml sets informational: false, or drops the statuses entirely, passes every mechanical check the fleet runs. Codecov then gates pull requests in that repo and nothing reports it until someone notices a blocked PR.

This is not specific to codecov.yml. It is a property of every intent-tier entry, and the question is which of them deserve a mechanical floor underneath the intent reading.

Why It Is Not Simply "Make It Letter Tier"

Intent tier is the correct choice for codecov.yml as a whole file. Each repo's ignore paths are its own, so a byte-match against the reference would be wrong by construction, and that is exactly why the entry is not verbatim.

What is mechanically checkable is narrower than the whole file: that coverage.status.project.default.informational and coverage.status.patch.default.informational are both true, where the file exists and the repo claims coverage. The ignore block stays a judgment call, since only a human or a reading agent knows which of a repo's paths are legitimately untested.

So the shape worth considering is a partial mechanical assertion inside an otherwise intent-tier file, which the fidelity model does not currently express. That is the actual design question.

Prior Art and Constraints

  • spec/audit.py currently evaluates no check belonging to a type in spec/project-types.json, and AUDIT.md states this in two places, with matching statements in audit.py itself. Any move here has to reconcile those, the same constraint noted in Audit check: an MTP-based dotnet test project with no global.json runner opt-in #1108.
  • codecov.yml is YAML, and the audit runs on the standard library with no YAML parser, so a check would need the same anchored-regex approach the Dependabot ecosystem check already uses, with its documented limits.
  • coverage_active already derives whether a repo claims coverage at build profile, so the gate for such a check exists.

Not Blocking

PR #1107 declined this as out of its scope, correctly: the codecov.yml requirement is pre-existing text that PR does not touch, and spec/files.json does already carry the file at intent fidelity against a reference containing both requirements. This issue captures the deeper point rather than the one that was declined.

Related: #1108, which needs the same AUDIT.md reconciliation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    gateA rule with no mechanical check, or a check that misses a shape

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions