Add a Coverage column, read from CI artifacts - #24
Merged
Conversation
Coverage comes from the `coverage-report` artifact CI already uploads, so the
number is tied to a commit and a branch. Two sources were rejected first:
* The .coverage files lying in the checkouts. 18 of 24 repos have one, aged
1 to 16 days. It records whenever someone last ran pytest in that
directory - possibly a subset of tests, possibly on a feature branch. It
would be a column that looks like a fact about the repo and is actually a
fact about a shell history.
* test LOC / code LOC. Cheap, complete, and not coverage. Fine as a ratio,
dishonest under that heading.
The branch filter is the load-bearing part. Artifacts come back newest-first
across every ref, and in a repo that tags releases the newest is usually a tag
build - rhiza's most recent coverage artifact is from v1.7.1, not main. Taking
the latest would report a release build's coverage as the repo's, measured at
a different commit. There is a test for exactly this.
Cost, measured rather than estimated
* Listing artifacts is one call per repo and always happens, so a report
published between refreshes is picked up: +25 calls, ~150/hour.
* The zip is downloaded only when the artifact id changes: 19 downloads on a
cold pass, 0 on the next.
That measurement also showed the documented API budget was wrong - the page
claimed 119 calls per refresh and ~1430/hour for a fleet of 32. Counting
requests through a full refresh gives 370 and ~2220/hour for a fleet of 25.
Corrected, with a note saying so.
Two metrics, not one. jq_ci_coverage_percent is the figure; jq_ci_coverage_lines
is what CI actually measured, and the percentage is not interpretable without
it. The denominator is NOT jq_local_code_lines: CI measures whatever it pointed
--cov at, so rhiza reads 100% of 176 lines while LOC counts 1477. Both are
right and they answer different questions.
Absent, never zero, when a repo publishes no report - six of this fleet do not,
and "nobody publishes coverage here" is not the finding "nothing is covered".
The column reads `no report`.
A malformed or oversized artifact is logged and skipped rather than failing a
refresh that has already gathered everything else, with guards on both the
download and the unpacked size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Coverage to Size and cadence, sourced from the
coverage-reportartifact your CI already uploads — so the number is tied to a commit and a branch.Two sources I rejected first
.coveragefiles in the checkouts. 18 of 24 repos have one, aged 1 to 16 days. It records whenever someone last ran pytest in that directory — possibly a subset of tests, possibly on a feature branch. A column that looks like a fact about the repo and is actually a fact about a shell history.The branch filter is load-bearing
Artifacts come back newest-first across every ref, and in a repo that tags releases the newest is usually a tag build — rhiza's most recent coverage artifact is from
v1.7.1, notmain. Taking the latest would report a release build's coverage as the repo's, measured at a different commit. Pinned by a test.Cost, measured rather than estimated
That measurement also caught a documentation bug: the API-budget page claimed 119 calls/refresh and ~1430/hour. Counting requests through a full refresh gives 370 and ~2220/hour. Corrected, with a note saying it was understated.
Two metrics, not one
jq_ci_coverage_percentis the figure.jq_ci_coverage_linesis what CI actually measured — and the percentage is not interpretable without it. The denominator is notjq_local_code_lines: CI measures whatever it pointed--covat, so rhiza reads 100% of 176 lines while the LOC column counts 1477. Both are right; they answer different questions.Absent, never zero, when a repo publishes no report — six of yours don't. The column reads
no report, because "nobody publishes coverage here" is not the finding "nothing is covered".What you'll actually see
All 19 reporting repos are at 100.0%. I verified that's real (
lines-covered == lines-validin every report), not a parsing artifact. So this column is a regression tripwire, not a differentiator — the varying, informative number is the line count, from 49 to 4,652. Worth knowing before you judge the column: if you'd rather see the denominator on the board too, say so and I'll add it as a second column.Checks
74 tests (15 new, incl. the branch trap, the download cache, malformed/oversized archives), ruff,
check-dashboard.py,mkdocs build --strict, compose parses. Verified live against all 25 repos.