You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Raised by CodeRabbit against PR #1107 and separated out from it, since the observation is about the fidelity model rather than about that PR's MTP migration.
The Observation
A requirement stated in WORKFLOW.md and carried in spec/files.json at fidelity: intent is evaluated by a reading agent against a catalog/ reference, never mechanically. Nothing fails when the repo drifts from it. The concrete case raised was codecov.yml:
D1.6 requires that file to set project and patch statuses to informational: true, so a coverage delta never gates a pull request, and to exclude intentionally-untested code from the denominator via ignore. The reference snippet carries both. But a repo whose codecov.yml sets informational: false, or drops the statuses entirely, passes every mechanical check the fleet runs. Codecov then gates pull requests in that repo and nothing reports it until someone notices a blocked PR.
This is not specific to codecov.yml. It is a property of every intent-tier entry, and the question is which of them deserve a mechanical floor underneath the intent reading.
Why It Is Not Simply "Make It Letter Tier"
Intent tier is the correct choice for codecov.yml as a whole file. Each repo's ignore paths are its own, so a byte-match against the reference would be wrong by construction, and that is exactly why the entry is not verbatim.
What is mechanically checkable is narrower than the whole file: that coverage.status.project.default.informational and coverage.status.patch.default.informational are both true, where the file exists and the repo claims coverage. The ignore block stays a judgment call, since only a human or a reading agent knows which of a repo's paths are legitimately untested.
So the shape worth considering is a partial mechanical assertion inside an otherwise intent-tier file, which the fidelity model does not currently express. That is the actual design question.
codecov.yml is YAML, and the audit runs on the standard library with no YAML parser, so a check would need the same anchored-regex approach the Dependabot ecosystem check already uses, with its documented limits.
coverage_active already derives whether a repo claims coverage at build profile, so the gate for such a check exists.
Not Blocking
PR #1107 declined this as out of its scope, correctly: the codecov.yml requirement is pre-existing text that PR does not touch, and spec/files.json does already carry the file at intent fidelity against a reference containing both requirements. This issue captures the deeper point rather than the one that was declined.
Related: #1108, which needs the same AUDIT.md reconciliation.
Raised by CodeRabbit against PR #1107 and separated out from it, since the observation is about the fidelity model rather than about that PR's MTP migration.
The Observation
A requirement stated in
WORKFLOW.mdand carried inspec/files.jsonatfidelity: intentis evaluated by a reading agent against acatalog/reference, never mechanically. Nothing fails when the repo drifts from it. The concrete case raised wascodecov.yml:{ "path": "codecov.yml", "fidelity": "intent", "reference": "catalog/snippets/configs/codecov.yml", "intentRef": "WORKFLOW.md", "appliesTo": ["csharp", "python"] }D1.6 requires that file to set project and patch statuses to
informational: true, so a coverage delta never gates a pull request, and to exclude intentionally-untested code from the denominator viaignore. The reference snippet carries both. But a repo whosecodecov.ymlsetsinformational: false, or drops the statuses entirely, passes every mechanical check the fleet runs. Codecov then gates pull requests in that repo and nothing reports it until someone notices a blocked PR.This is not specific to
codecov.yml. It is a property of every intent-tier entry, and the question is which of them deserve a mechanical floor underneath the intent reading.Why It Is Not Simply "Make It Letter Tier"
Intent tier is the correct choice for
codecov.ymlas a whole file. Each repo'signorepaths are its own, so a byte-match against the reference would be wrong by construction, and that is exactly why the entry is notverbatim.What is mechanically checkable is narrower than the whole file: that
coverage.status.project.default.informationalandcoverage.status.patch.default.informationalare bothtrue, where the file exists and the repo claims coverage. Theignoreblock stays a judgment call, since only a human or a reading agent knows which of a repo's paths are legitimately untested.So the shape worth considering is a partial mechanical assertion inside an otherwise intent-tier file, which the fidelity model does not currently express. That is the actual design question.
Prior Art and Constraints
spec/audit.pycurrently evaluates no check belonging to a type inspec/project-types.json, andAUDIT.mdstates this in two places, with matching statements inaudit.pyitself. Any move here has to reconcile those, the same constraint noted in Audit check: an MTP-based dotnet test project with no global.json runner opt-in #1108.codecov.ymlis YAML, and the audit runs on the standard library with no YAML parser, so a check would need the same anchored-regex approach the Dependabot ecosystem check already uses, with its documented limits.coverage_activealready derives whether a repo claims coverage at build profile, so the gate for such a check exists.Not Blocking
PR #1107 declined this as out of its scope, correctly: the
codecov.ymlrequirement is pre-existing text that PR does not touch, andspec/files.jsondoes already carry the file at intent fidelity against a reference containing both requirements. This issue captures the deeper point rather than the one that was declined.Related: #1108, which needs the same
AUDIT.mdreconciliation.