feat: testgraph registers itself, so the ledger has something to record - #59
Merged
Conversation
The results ledger shipped in #57 and held zero rows. Not a bug — an absence of opportunity. `hooks/install.sh` installs only into repos with an approved registry, and the only two were honeyslate (last pushed 2026-07-17) and signedintake (2026-07-19). testgraph is the most actively pushed repo here and was the one repo the hook did not cover. `propose --repo .` correctly wrote nothing: 0 route handlers, and it named the reason from its own blind-spot list (CLI entry points). So the registry is hand-authored — one journey per CLI surface, 19 entry nodes, all six resolving with no approval or drift warning. Selectivity, measured with Update 3's methodology correction applied from the start (`harness/selectivity.py`, fresh per-commit index, 20 commits of main): mean 2.07/6 (34.4%) over the 15 commits that touch product code, 65.6% of journey-runs avoided. Histogram {0:5, 1:8, 2:4, 3:1, 6:2}. All five zero-selections are exactly the five non-product commits, and every product-touching commit selected at least one journey. I got this wrong once mid-task and the correction is worth keeping: two hand-picked ranges each named 6/6, and I concluded selectivity had collapsed on a small tightly-coupled codebase. The sweep says otherwise — 34.4% sits beside signedintake's 27.7%. The two ranges were unrepresentative. The harness exists because that intuition is unreliable; the registry note carried the wrong claim briefly and was rewritten. What this is NOT evidence for: instrument and target are one repo. Selectivity survives that better than recall, since the harness never sees the registry's authorship, but observed_recall from the ledger will be testgraph scoring testgraph. Treat the ledger's testgraph rows as a liveness test of the selection -> outcome join, not as an accuracy claim. The accuracy claims in TECHNICAL.md still come from honeyslate's hand labels and the outside repos. `tests/test_registries.py` closes a gap nothing covered: no test validated any shipped registry, which is exactly the ae128bb finding-8 defect — an unparseable registry reads as "no registry", so the hook logs NO_REGISTRY, exits 0, and a typo silently disables the tool until somebody notices it stopped talking. Index-free by design (honeyslate's and signedintake's codegraph dbs are not present in every checkout). Both guards mutation-tested: an unparseable file and a duplicate target each fail the suite. Also recorded in TECHNICAL.md, because it blocked this work entirely: codegraph cannot find a bare-worktree repo root. A worktree's `.git` is a file, not a directory, so codegraph 1.5.0 walks up past the repo — and `codegraph index <path>` rebuilds whichever ancestor index it finds rather than creating a local one. A 279 MB stray index at ~/personal_projects had been answering every query from any un-indexed repo under that root. Use `codegraph init <path>` for a first index, never `codegraph index <path>`. 243 tests (was 238).
Five review findings. The first falsifies a claim this PR published.
1. `testgraph/db.py` and `testgraph/registry.py` are reachable from NO journey.
Seeding every node in either file and running the full impacted closure
intersects the entry map in zero nodes: db.py's closure is 14 nodes and never
leaves the file, registry.py's is 24, against ledger.py's 257 -> all six
journeys. So a commit rewriting the whole of db.py — the graph traversal this
tool IS — answers `journeys to test: NONE` with status OK and
recall_degraded false.
The 20-commit window could not see it because no commit in it touches either
file alone, which makes the "zero silent nothing-affected answers" line in
Update 5 an artifact of co-changed files. That line is now removed and
replaced with the measurement that refutes it.
The cause is not the registry and cannot be fixed there: codegraph records no
cross-file edges for ALIASED relative imports. Cross-file inbound edge counts
are exactly 0 for the two modules imported `from . import X as Y`, versus 15
for integrity, 44 for ledger and 15 for hook, all unaliased. Adding these
files' symbols as journey entries would manufacture the right answer from a
false claim about what an entry is, so the registry declares the gap instead.
A NONE after a db.py or registry.py change means UNKNOWN until the indexer
links aliased imports.
2. No `spot_checks`, so `integrity.check` received `{}` and the caller-count
check silently no-opped — the one check of the three that catches the
2026-07-17 incident integrity.py exists for, and the one `codegraph sync`
cannot clear. Added `append` (>=5, live 7) and `StampError` (>=4, live 6),
the two propose nominated, and a test asserting EVERY shipped registry
carries them.
3. `test_testgraph_registry_is_resolvable_for_this_repo` passed or failed on
what the checkout directory is CALLED, because `repo_name` derives the target
from the directory name. It failed in any worktree not literally named
`testgraph` — including every scratch worktree selectivity.py and accuracy.py
create, which is exactly where it runs once testgraph is its own measured
target. The repo path is now synthesized in a temp dir. Verified: the suite
is green from a directory named `notthesamename`, which previously failed.
4. The `sorted(journeys, key=journey_sort_key)` guard was vacuous — the key
cannot raise for any str, so it stayed green under any mutation. It now
asserts the property the renderer depends on: J2 before J10.
5. Known Limitations still said honeyslate was the only approved registry,
contradicting Update 5 thirty-five lines above and signedintake.json. Both it
and the USAGE.md FAQ now say three, and name honeyslate as the only one with
hand labels — so the only one with a recall figure at all.
244 tests (was 243).
Two corrections to the claim made one commit ago. 1. "Cross-file inbound edge counts are exactly 0 for aliased imports" was too strong. `select.py` is also aliased (`from . import select as sel`) and has one edge, not zero. The claim was literally true of the two modules it named but implied a clean dichotomy that does not hold, and a reader checking select.py would have found the counter-example before I did. Restated as the measured table across all five same-package modules called as `module.attr()`: registry.py 38 call sites -> 0 edges, db.py 18 -> 0, select.py 6 -> 1, ledger.py 10 -> 44, integrity.py 3 -> 15. The mechanism is sharper than "aliased imports" too: codegraph does not resolve calls through a module bound under an alias, because the local name no longer matches the module name. Same import form, same call form, alias the only difference. 2. Checked whether this reaches honeyslate, since honeyslate is the only repo whose numbers are quoted as ACCURACY rather than sizes. It does not. honeyslate has 9 aliased imports and every one aliases a symbol (`Session as DbSession`), never a module, so the `alias.attr()` form never arises. The recall and precision figures in TECHNICAL.md are unaffected, and that is now recorded where the limitation is described rather than left for the next reader to re-derive. 244 tests.
This was referenced Aug 6, 2026
inth3shadows
added a commit
that referenced
this pull request
Aug 6, 2026
…#61) `hooks/install.sh` was committed mode 100644. README.md and USAGE.md both document running it directly: hooks/install.sh # every repo with an approved registry hooks/install.sh /path/to/repo # or just one hooks/install.sh --uninstall All three fail with "Permission denied". Found by following the documentation while installing the hook into testgraph itself after #59 merged — the whole point of that PR was to close the gap where nothing ever called the selector, and the documented way to start calling it did not run. It went unnoticed because every previous invocation was `bash hooks/install.sh`, which ignores the mode bit entirely, so the file's own tests and every manual run passed while the documented path did not. `hooks/pre-push` is deliberately left 100644: it is a template that gets sed'd into the hook file, and install.sh chmods the OUTPUT. Nothing executes the template in place. No behaviour change beyond the mode bit; 251 tests unaffected.
This was referenced Aug 6, 2026
inth3shadows
added a commit
that referenced
this pull request
Aug 14, 2026
* measure: trace-derived ground truth against testgraph itself (#12) honeyslate's Postgres is still unreachable (port :55433 closed, no Docker under WSL), so this runs the cheaper target Update 4 proposed: testgraph's own 251-test stdlib suite against its own registry from #59. pytest runs the existing unittest suite unmodified. New harness/journey_tests_testgraph.json maps each tests/test_*.py file to a journey, checked against which module each file exercises UNMOCKED (test_hook patches select/registry on every path; test_core calls select/export for real). 81 traced symbols land outside the static footprint across all 6 journeys. 62 are the module-alias blind spot Update 5 already declared, now confirmed by an independent method. 8 are this map's own file-level over-approximation noise (test_core.py double-mapped to J2 and J4). 11 are new: ledger.append has zero real inbound static edges despite unaliased call syntax identical to ledger.path two lines away, while 7 unrelated .append() list calls elsewhere get bare-name-matched to it instead — which is why the registry's own min_caller_edges spot-check passed while pointing at the wrong callers entirely. A commit that breaks ledger.append currently selects neither J1 nor J5. Full writeup in TECHNICAL.md Update 7. Not fixed here — the edge-resolution defect is in CodeGraph's resolver, a distinct root cause from the already-documented alias gap, and gets its own issue. * fix: correct Update 7's traced_only bucket counts, caught by review Grouped by file totals instead of (journey, symbol) pairs, which hid two real errors that happened to cancel in the sum: select.py was undercounted (1 claimed vs 6 actual) and ledger.py was overcounted (11 claimed vs 4 actual, even though the sentence's own enumeration — "append x2, path, state_dir" — already summed to 4). harness/plugin/tgtrace.py's 2 traced-only symbols were omitted from every bucket entirely. Re-derived from the ground_truth --json output grouped by (journey, symbol): 63 alias blind spot (the 62 already known, plus one more instance in select.py via propose.py's sel._is_test call — same aliased-module pattern, not previously listed), 12 map-granularity noise (two distinct mechanisms, not one), 2 the tgtrace.py blind spot the registry's own note already declares, 4 the ledger.append edge defect (#66). 63+12+2+4=81. * fix: ledger.path is one line after ledger.append, not two
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the reason the results ledger from #57 has held zero rows since it merged.
Why there were no rows
Not a bug — an absence of opportunity.
hooks/install.shinstalls only into repos with an approved registry, and the only two were honeyslate (last pushed 2026-07-17) and signedintake (2026-07-19). Both dormant. Meanwhile testgraph is the most actively pushed repo on this machine and was the one repo the hook did not cover.What's here
journeys/testgraph.json— hand-authored,approved: true, 6 journeys (J1 hook, J2 select, J3 propose, J4 export, J5 record, J6 harness), 19 entry nodes, all resolving, no approval or drift warning.propose --repo .correctly wrote nothing:found 0 route handler(s), and it named the reason from its own blind-spot list — CLI entry points and management commands. A proposer that had invented six journeys from nothing would have been the silent-confidence failure this codebase keeps designing against.tests/test_registries.py— 5 tests over every shipped registry. Closes a gap nothing covered.TECHNICAL.md Update 5 — the measurement and its caveats.
Selectivity
harness/selectivity.py, fresh per-commit index, last 20 commits ofmain. Update 3's methodology correction applied from the start rather than retrofitted:<= 2journeysrecall_degradedHistogram over all 20:
{0: 5, 1: 8, 2: 4, 3: 1, 6: 2}.All five zero-selections are exactly the five non-product commits, and every one of the fifteen that touches product code selected at least one journey.
A wrong call, corrected mid-task
I first generalised from two hand-picked ranges that each named 6/6 journeys and concluded selectivity had collapsed on a small tightly-coupled codebase. The sweep says otherwise — 34.4% sits beside signedintake's 27.7%, not below it. The two ranges were unrepresentative: one a cross-cutting ledger commit, one containing a file with no indexed symbols. The registry note briefly carried the wrong claim and was rewritten. The harness exists precisely because that intuition is unreliable.
The populated middle is not a counter-example
testgraph's histogram occupies 1, 2 and 3 where mealie's was
{0: 38, 23: 2}. That is not evidence against Update 3's bimodality finding: with six journeys the "middle" is only 1–5 wide, so a registry this small cannot exhibit the shape either way. Noted in TECHNICAL.md so it isn't later mistaken for one.What this is not evidence for
The instrument and the target are the same repo. Selectivity survives that better than recall —
harness/selectivity.pychecks each commit into its own worktree, builds its own index, and never sees who wrote the registry — butobserved_recallfrom the ledger will be testgraph scoring testgraph against a registry authored while reading testgraph's source.Treat the ledger's testgraph rows as a liveness test of the selection → outcome join, not as an accuracy claim. The accuracy claims in TECHNICAL.md still come from honeyslate's hand labels and the outside repos in Update 3, and nothing here changes them.
The registry test closes a real gap
Nothing validated any shipped registry. That is exactly the
ae128bbfinding-8 defect: a trailing comma makesjson.loadraise,reg.resolve_for_reposwallows theValueErrorand returnsNone, the hook logsNO_REGISTRYand exits 0, andrecordtells you to draft a file you already have. A typo silently disables the tool for that repo until somebody notices it stopped talking.Index-free by design — honeyslate's and signedintake's codegraph databases are not present in every checkout, so asserting their entries resolve would fail for reasons unrelated to the registry.
Both guards mutation-tested rather than assumed green (the repo's own standard, after five of seven skill-contract guards once passed against the defect they named):
journeys/FAILED (failures=1, errors=1)testgraphFAILED (failures=2)OKOperational finding: codegraph cannot find a bare-worktree repo root
This blocked the work entirely and is worth knowing beyond this repo.
In the
claudewlayout a worktree's.gitis a file (gitdir: .../.bare/worktrees/main), not a directory. codegraph 1.5.0's root detection wants a directory, so from any worktree it walks up past the repo.codegraph index <path>then rebuilds whichever ancestor index it finds rather than creating a local one — or refuses at$HOMEwhen there is none.A 279 MB
~/personal_projects/.codegraph(11,141 files, 55 repos, truncated mid-run with 880,880 unresolved references) had been sitting there answering every query from any un-indexed repo under that root. It had to be removed before testgraph could be indexed at all.Use
codegraph init <path>for a first index, nevercodegraph index <path>—initcreates in the directory named,indexresolves an ancestor.Not in this PR
Success criteria 3 and 4 are gated on merge:
hooks/install.shbakesTESTGRAPH_HOME=.../testgraph/main, so the registry has to be onmainbefore the hook can see it — the installer's own note says to install after merging. Once merged: install the hook, push, and confirm astatus: OKselection row lands in the ledger.Verification
243 tests green, was 238.