feat: the ledger gets the half nothing could write — journey outcomes (#10) - #57
Merged
Conversation
The pre-push hook (#51) gave the selector a caller, so `selection` rows come for free. This adds `outcome` rows — what running a journey then FOUND — and the join between them, which is the only thing either kind is worth. Joined on (repo, commit): a journey that failed on a commit whose selection did not name it is a silent under-selection, the one failure mode a recall-first selector must not have. A failure on a commit testgraph was never asked about is counted apart, as `unjudged` — collapsing the two would score every failure recorded before the hook was installed as a recall miss. Storage declines #10's "put it in the KB" at the write path, and honours it at the read path. The KB is reachable only through an MCP an agent drives; there is no Python client, so hook.py cannot be the writer without a tunnel and a credential inside a hook whose contract is that it never fails a push. A KB-only ledger has no writer at all — the same defect #8 named, one layer up. So: local JSONL writes, `record --summary --export-kb` hands an agent a payload to propose. One of #10's three reasons for rejecting a local store was already false — state_dir() has always been outside every worktree. Ranking is deliberately NOT wired to this. The ledger held zero rows when this shipped; the hook has not fired once since it merged. `ready_for_ranking` gates at 20 judged commits, the size of the seeded-regression eval. invocations.jsonl is still read as `selection` rows, so no install loses its history to the rename.
Nine review findings, two of which corrupted the one number this ledger exists
to produce. Both had the same shape: a row that says nothing about the selector
counted as evidence against it.
1. HIGH. `hook.run` writes selection rows on four NON-ANSWER paths —
NO_REGISTRY, NO_INDEX, ERROR, BLOCKED — each with no journey_ids. The join
read those as "asked, and it named nothing", so a tripped integrity guard
scored every later failure at that commit as a silent miss. It chained: one
JSON typo in an approved registry makes resolve_for_repo return None, so
every push logs NO_REGISTRY and every failure after it is the selector's
fault. Only status == OK now counts as an answer.
2. HIGH. The join compared a DELTA against a STATE. A selection answers "what
could base..head break"; an outcome asserts "J is broken AT this commit".
Joining on head alone: A..B breaks J3 and B's selection NAMES it; nobody runs
journeys; B..C touches only README so C's selection names nothing; the
developer runs J3 at HEAD (=C) exactly as USAGE.md says, it fails, recorded
at C -> silent miss. The selector was right both times.
Fixed by requiring a green baseline: a failure is only `missed` if a `pass`
is recorded for that journey at the push's BASE. Otherwise `unbaselined`.
You only get to blame the selector when you had something to regress from.
Misses stay rare unless journeys run per push — that discipline is what the
number needs to mean anything, so the cost is the point.
`unjudged` accordingly splits into `unasked` (nothing answered) and
`unbaselined` (answered, no baseline). Merging them would be the same defect
this file refuses everywhere else.
3. resolve_commit returned the rev VERBATIM when git could not answer, so
`record --repo <the bare-worktree parent>` — a directory whose repo_name
still resolves, so every other check passes — filed every outcome under the
key "HEAD", where unrelated commits joined to each other: one fabricated
catch and one fabricated miss from a single key. Returns None; record
refuses; the join accepts only 40-hex shas.
4. caught/missed counted outcome ROWS, so re-recording one failure multiplied
the headline while judged_commits stayed deduplicated — the two numbers
disagreed. Deduped on (commit, journey), last verdict wins.
5. The ranking gate was satisfiable by 20 `skip` rows, i.e. by journeys
explicitly not run. Needs judged commits AND a judged failure; skip counts
toward neither.
6. read() promised a malformed row is skipped, then raised TypeError sorting a
well-formed row with a string `ts` — and the KB export loop invites an agent
to write into this file.
7. `rev-parse <tag>` returned the tag OBJECT oid for an annotated tag, a key no
selection can carry. Uses ^{commit}.
8. An unparseable registry was reported as "no registry — draft one", telling
the user to write a file they have, and hiding that the same typo silently
disabled the pre-push hook.
9. The legacy-migration test supplied a `commit` key real legacy rows never
had, so the head-sha fallback it exists to protect was untested.
212 tests (was 203). The honeyslate smoke run that previously reported "1 SILENT
MISS" — a false positive from finding 2 — now correctly reports it unbaselined
and says why.
Owner
Author
Review fixes —
|
inth3shadows
added a commit
that referenced
this pull request
Aug 6, 2026
…rd (#59) * feat: testgraph registers itself, so the ledger has something to record The results ledger shipped in #57 and held zero rows. Not a bug — an absence of opportunity. `hooks/install.sh` installs only into repos with an approved registry, and the only two were honeyslate (last pushed 2026-07-17) and signedintake (2026-07-19). testgraph is the most actively pushed repo here and was the one repo the hook did not cover. `propose --repo .` correctly wrote nothing: 0 route handlers, and it named the reason from its own blind-spot list (CLI entry points). So the registry is hand-authored — one journey per CLI surface, 19 entry nodes, all six resolving with no approval or drift warning. Selectivity, measured with Update 3's methodology correction applied from the start (`harness/selectivity.py`, fresh per-commit index, 20 commits of main): mean 2.07/6 (34.4%) over the 15 commits that touch product code, 65.6% of journey-runs avoided. Histogram {0:5, 1:8, 2:4, 3:1, 6:2}. All five zero-selections are exactly the five non-product commits, and every product-touching commit selected at least one journey. I got this wrong once mid-task and the correction is worth keeping: two hand-picked ranges each named 6/6, and I concluded selectivity had collapsed on a small tightly-coupled codebase. The sweep says otherwise — 34.4% sits beside signedintake's 27.7%. The two ranges were unrepresentative. The harness exists because that intuition is unreliable; the registry note carried the wrong claim briefly and was rewritten. What this is NOT evidence for: instrument and target are one repo. Selectivity survives that better than recall, since the harness never sees the registry's authorship, but observed_recall from the ledger will be testgraph scoring testgraph. Treat the ledger's testgraph rows as a liveness test of the selection -> outcome join, not as an accuracy claim. The accuracy claims in TECHNICAL.md still come from honeyslate's hand labels and the outside repos. `tests/test_registries.py` closes a gap nothing covered: no test validated any shipped registry, which is exactly the ae128bb finding-8 defect — an unparseable registry reads as "no registry", so the hook logs NO_REGISTRY, exits 0, and a typo silently disables the tool until somebody notices it stopped talking. Index-free by design (honeyslate's and signedintake's codegraph dbs are not present in every checkout). Both guards mutation-tested: an unparseable file and a duplicate target each fail the suite. Also recorded in TECHNICAL.md, because it blocked this work entirely: codegraph cannot find a bare-worktree repo root. A worktree's `.git` is a file, not a directory, so codegraph 1.5.0 walks up past the repo — and `codegraph index <path>` rebuilds whichever ancestor index it finds rather than creating a local one. A 279 MB stray index at ~/personal_projects had been answering every query from any un-indexed repo under that root. Use `codegraph init <path>` for a first index, never `codegraph index <path>`. 243 tests (was 238). * fix: the registry claimed a clean sweep it had not earned Five review findings. The first falsifies a claim this PR published. 1. `testgraph/db.py` and `testgraph/registry.py` are reachable from NO journey. Seeding every node in either file and running the full impacted closure intersects the entry map in zero nodes: db.py's closure is 14 nodes and never leaves the file, registry.py's is 24, against ledger.py's 257 -> all six journeys. So a commit rewriting the whole of db.py — the graph traversal this tool IS — answers `journeys to test: NONE` with status OK and recall_degraded false. The 20-commit window could not see it because no commit in it touches either file alone, which makes the "zero silent nothing-affected answers" line in Update 5 an artifact of co-changed files. That line is now removed and replaced with the measurement that refutes it. The cause is not the registry and cannot be fixed there: codegraph records no cross-file edges for ALIASED relative imports. Cross-file inbound edge counts are exactly 0 for the two modules imported `from . import X as Y`, versus 15 for integrity, 44 for ledger and 15 for hook, all unaliased. Adding these files' symbols as journey entries would manufacture the right answer from a false claim about what an entry is, so the registry declares the gap instead. A NONE after a db.py or registry.py change means UNKNOWN until the indexer links aliased imports. 2. No `spot_checks`, so `integrity.check` received `{}` and the caller-count check silently no-opped — the one check of the three that catches the 2026-07-17 incident integrity.py exists for, and the one `codegraph sync` cannot clear. Added `append` (>=5, live 7) and `StampError` (>=4, live 6), the two propose nominated, and a test asserting EVERY shipped registry carries them. 3. `test_testgraph_registry_is_resolvable_for_this_repo` passed or failed on what the checkout directory is CALLED, because `repo_name` derives the target from the directory name. It failed in any worktree not literally named `testgraph` — including every scratch worktree selectivity.py and accuracy.py create, which is exactly where it runs once testgraph is its own measured target. The repo path is now synthesized in a temp dir. Verified: the suite is green from a directory named `notthesamename`, which previously failed. 4. The `sorted(journeys, key=journey_sort_key)` guard was vacuous — the key cannot raise for any str, so it stayed green under any mutation. It now asserts the property the renderer depends on: J2 before J10. 5. Known Limitations still said honeyslate was the only approved registry, contradicting Update 5 thirty-five lines above and signedintake.json. Both it and the USAGE.md FAQ now say three, and name honeyslate as the only one with hand labels — so the only one with a recall figure at all. 244 tests (was 243). * fix: state the alias mechanism as measured, and clear honeyslate Two corrections to the claim made one commit ago. 1. "Cross-file inbound edge counts are exactly 0 for aliased imports" was too strong. `select.py` is also aliased (`from . import select as sel`) and has one edge, not zero. The claim was literally true of the two modules it named but implied a clean dichotomy that does not hold, and a reader checking select.py would have found the counter-example before I did. Restated as the measured table across all five same-package modules called as `module.attr()`: registry.py 38 call sites -> 0 edges, db.py 18 -> 0, select.py 6 -> 1, ledger.py 10 -> 44, integrity.py 3 -> 15. The mechanism is sharper than "aliased imports" too: codegraph does not resolve calls through a module bound under an alias, because the local name no longer matches the module name. Same import form, same call form, alias the only difference. 2. Checked whether this reaches honeyslate, since honeyslate is the only repo whose numbers are quoted as ACCURACY rather than sizes. It does not. honeyslate has 9 aliased imports and every one aliases a symbol (`Session as DbSession`), never a module, so the `alias.attr()` form never arises. The recall and precision figures in TECHNICAL.md are unaffected, and that is now recorded where the limitation is described rather than left for the next reader to re-derive. 244 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10.
The pre-push hook (#51) gave the selector a caller, so
selectionrows come for free. This adds the half nothing could write:outcomerows — what running a journey then found — plus the join between them, which is the only thing either kind is worth on its own.The number this exists to produce
Joined on
(repo, commit):Keeping
unjudgedout ofmissedis the point. Collapsing them would score every failure recorded before the hook was installed as a recall miss.observed_recallisNone, not0.0, until something is actually judged.Two decisions that differ from the issue — both argued in TECHNICAL.md
1. Storage: local JSONL writes, KB via export. #10 decided the ledger belongs in the shared
kb.*Postgres, not a local store. Declined at the write path:hook.pycannot be the writer without a tunnel, a credential and a network round-trip inside a hook whose entire contract is that it never fails a push.state_dir()resolves to~/.local/share/testgraph, outside every worktree.wtcleanwas never a risk.The two reasons that stand — invisibility from the work Mac, no reviewer surface — are what
record --summary --export-kbcloses: it emits a payload an agent proposes. It deliberately does not name a target table, since KB conventions requirekb.read.searchfirst andkb.propose.extendover a new table.2. Ranking is not wired to this. #10 asks for failure history to feed the next ranking. It should — but the ledger held zero rows when this shipped; the hook has not fired once since it merged.
ready_for_rankinggates atMIN_JUDGED_COMMITS = 20, the size of the seeded-regression eval (#5) — the smallest set that has said anything falsifiable about this selector. Below it,--summaryprints the distance rather than inventing a signal.Also
invocations.jsonlis still read, asselectionrows, so no install loses its history to the rename.Verification
python3 -m unittest discover -s tests— 203 tests, OK (182 existing + 21 new). Smoke-tested end to end against honeyslate: a realhookrun plus a recorded failure at that commit correctly reports 1 silent miss.