Skip to content

coverage: every codecov status is informational, so a component sitting below its own target still produces a green build #402

Description

@Yaraslaut

What is wrong

Every coverage status in codecov.yml — the two defaults and all eight
components — carries informational: true. A component below its own target
reports the number and passes.

Two components are below their own target on master right now:

component target in codecov.yml measured on master
ladder (examples/common) 95% 90.39%
crm (rung 7) 89% 87.67%

examples/common is the worse of the two in a way the number alone hides:
codecov.yml's own comment records the target being derived from a measurement
of 95.85%, with a documented ceiling of 98.56%. It is now 90.39%. That is a
5.46-point regression that no check reported, in the component the file
describes as carrying "the whole of that promise today".

The second half: the library has no component at all

component_management names eight components, and every one of them is under
examples/. include/morph — the framework, 5,716 of the report's 15,546
lines — is scored only by the default status, which has no target and is
informational: true. So the product itself is gated by nothing, and no number
anywhere states what its coverage is supposed to be.

Measured on master, include/morph is at 95.69% (5,470/5,716 lines;
126 misses, 120 partials).

The contradiction this creates

examples/IMPLEMENTATION.md's rung-exit criteria name something that does not
exist:

  1. Model coverage gate green (rule 5) — the blocking codecov.yml
    component, target set from a measured ceiling with every known-artifact
    line documented, matrix green.

No component in codecov.yml blocks. A rung can satisfy exit criterion 5 by
passing a gate that cannot fail — which is invariant 7 in the shape this
repository keeps finding: a control that appears to work while measuring
nothing.

The constraint that makes the naive fix wrong

Flipping every informational: true to false today would immediately block
merges on two components that are already red, and it would contradict a
recorded decision. codecov.yml's comments state the policy explicitly:

These rungs are below where they should be, and a blocking gate today would
stop work on rungs 5-8 to pay down debt from rungs 2-4 — the wrong order. The
decision is: finish building the ladder, then raise these.

examples/rungs.txt now lists seven rungs through crm, so that condition is
near but its resolution is the repo owner's, not a triage pass's.

Second constraint: several components document a measured ceiling below
100%
— llvm-cov places a "control reached past this block" counter on certain
closing braces and there is no LCOV_EXCL_LINE equivalent to suppress it. Any
blocking target must sit below its component's measured ceiling rather than at
an aspiration, or it fails on an artifact.

Verification status: reproduced

Measured against the Codecov API for LASTRADA-Software/morph, branch
master:

GET /api/v2/github/LASTRADA-Software/repos/morph/components/?branch=master
  ladder 90.39 | pastebin 95.81 | bookmarks 86.83 | polls 90.63
  kanban 85.72 | ledger 87.68  | lims 97.78      | crm 87.67

GET /api/v2/github/LASTRADA-Software/repos/morph/totals/?branch=master
  lines 15546  hits 14062  misses 851  partials 633  coverage 90.45

Targets read from codecov.yml on master; the informational: true on every
status is verifiable by reading the same file.

Not verified: whether Codecov currently considers the file valid. It was
invalid and wholly inert until morph#133; nothing re-checks that, and a config
that fails validation is discarded wholesale — which would make every finding
above true for a second, worse reason. Whoever takes this should re-run
curl -X POST --data-binary @codecov.yml https://codecov.io/validate as step
one.

What would close this

Close this when all three hold:

  1. codecov.yml and examples/IMPLEMENTATION.md agree with each other —
    either every rung component's project status is informational: false, or
    criterion 5 stops calling the gate blocking. Today they contradict.
  2. include/morph has a component of its own, with a target set from a
    measured ceiling the way every ladder component's is, so the framework is
    scored against a stated number rather than none.
  3. A pull request that drops a gated component below its target actually
    fails a required check
    — demonstrated on a real PR, not asserted from the
    config. A gate nobody has seen fail is the thing this issue is about.

Not closable by raising a target to meet current coverage. That deletes the
finding rather than the gap, and examples/common's 5.46-point regression is
the evidence that it would.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ciSubsystem: cibugSomething isn't workingtriage: validWell-framed; implement as written

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions