Add a test suite and CI that catch the bugs this repo has shipped - #4
Merged
Conversation
Three bugs reached main: an undefined Config attribute that crashed every GitHub refresh, deleted and renamed workflows counted as live, and a paginated runs feed that hid failing jobs. CodeQL - the only check here - passed on all three, because it never executes this code. The suite targets exactly those failure modes rather than chasing coverage: - test_config walks the AST for every `cfg.<attr>` the package reads and asserts Config defines it. That catches the whole class, not just the one attribute that bit us, and it would have failed on the commit that shipped it. - test_workflows drives the GitHub client against canned responses: deleted and disabled workflows ignored, a renamed workflow yielding one series under its current name, "latest" decided by completion time rather than feed order, workflows missing from the feed fetched directly, workflows that never ran absent rather than assumed green, an unreadable listing degrading to over-reporting, and two workflows sharing a name getting distinct labels. - test_metrics asserts the exposition is valid and never overstates health: a repo is green only when every workflow is, and no duplicate label sets are emitted. Each was verified by reintroducing the original bug and watching the relevant test fail - 6, 2 and 1 failures respectively. That last test found a live weakness. render() trusted github.py to hand it unique workflow names; when that invariant broke upstream, Prometheus silently dropped 16 samples per scrape. This layer owns the exposition contract, so it now collapses duplicates itself, newest run winning. CI runs lint, format, the suite, the dashboard validator, a check that the generated public dashboard is not stale, and compose validation. The overlays are validated as overlays because they only patch services and are not valid alone; the base file needs GITHUB_TOKEN set to anything to interpolate. Every step was dry-run locally before being written down. ruff's line-length moves into pyproject so `check` and `format` cannot disagree - passing the flag to one and not the other silently fails CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pure reformatting, no behaviour change: the package had been formatted at ruff's default 88 while lint ran at 100, so the first `ruff format --check` in CI would have failed on untouched files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| # Config field defaults bind at import, so re-evaluate the expression. | ||
| import importlib | ||
|
|
||
| import jq_collector.config as mod |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Three bugs reached
mainin short order:Configattribute that crashed every GitHub refresh (Fix broken main: define Config.public_only, and add a reviewed path to public serving #1)CodeQL — the only check on this repo — passed on all three, because it never executes this code. Every one was caught by hand.
What the suite covers
Targeted at exactly those failure modes rather than chasing coverage:
test_configwalks the AST for everycfg.<attr>the package reads and assertsConfigdefines it. That catches the whole class, not just the attribute that bit us.test_workflowsdrives the GitHub client against canned responses: deleted and disabled workflows ignored; a renamed workflow yielding one series under its current name; "latest" decided by completion time, not feed order; workflows missing from the feed fetched directly; workflows that never ran absent rather than assumed green; an unreadable listing degrading to over-reporting; two workflows sharing a name getting distinct labels.test_metricsasserts the exposition is valid and never overstates health: a repo is green only when every workflow is, and no duplicate label sets are emitted.No test touches the network.
Verified by reintroducing each bug
Config.public_onlyremovedIt already found something
render()trustedgithub.pyto hand it unique workflow names. When that invariant broke upstream, Prometheus silently dropped 16 samples per scrape ("samples with different value but same timestamp"— visible only in its logs, target still "up"). This layer owns the exposition contract, so it now collapses duplicates itself, newest run winning.The workflow
Lint, format, the suite, the dashboard validator, a staleness check on the generated public dashboard, and compose validation.
Two things worth noting, both found by dry-running the steps locally before writing them down:
GITHUB_TOKENset to anything, or interpolation failsruff'sline-lengthmoves intopyproject.tomlsocheckandformatcannot disagree — passing the flag to one and not the other silently fails CI. The second commit is the resulting reformat: pure formatting, no behaviour change, separated so it can be skimmed.Full pipeline dry-run from a clean
git archiveof this branch, on top of currentmain: all steps green, 20 tests passing.🤖 Generated with Claude Code