Skip to content

feat: testgraph registers itself, so the ledger has something to record - #59

Merged
inth3shadows merged 3 commits into
mainfrom
claude-20260806-084328
Aug 6, 2026
Merged

feat: testgraph registers itself, so the ledger has something to record#59
inth3shadows merged 3 commits into
mainfrom
claude-20260806-084328

Conversation

@inth3shadows

Copy link
Copy Markdown
Owner

Closes the reason the results ledger from #57 has held zero rows since it merged.

Why there were no rows

Not a bug — an absence of opportunity. hooks/install.sh installs only into repos with an approved registry, and the only two were honeyslate (last pushed 2026-07-17) and signedintake (2026-07-19). Both dormant. Meanwhile testgraph is the most actively pushed repo on this machine and was the one repo the hook did not cover.

What's here

  • journeys/testgraph.json — hand-authored, approved: true, 6 journeys (J1 hook, J2 select, J3 propose, J4 export, J5 record, J6 harness), 19 entry nodes, all resolving, no approval or drift warning.

    propose --repo . correctly wrote nothing: found 0 route handler(s), and it named the reason from its own blind-spot list — CLI entry points and management commands. A proposer that had invented six journeys from nothing would have been the silent-confidence failure this codebase keeps designing against.

  • tests/test_registries.py — 5 tests over every shipped registry. Closes a gap nothing covered.

  • TECHNICAL.md Update 5 — the measurement and its caveats.

Selectivity

harness/selectivity.py, fresh per-commit index, last 20 commits of main. Update 3's methodology correction applied from the start rather than retrofitted:

all 20 commits 15 touching product code
mean selected 1.55 / 6 (25.8%) 2.07 / 6 (34.4%)
journey-runs avoided 74.2% 65.6%
<= 2 journeys 17 of 20 12 of 15
recall_degraded 1 of 20 1 of 15

Histogram over all 20: {0: 5, 1: 8, 2: 4, 3: 1, 6: 2}.

All five zero-selections are exactly the five non-product commits, and every one of the fifteen that touches product code selected at least one journey.

A wrong call, corrected mid-task

I first generalised from two hand-picked ranges that each named 6/6 journeys and concluded selectivity had collapsed on a small tightly-coupled codebase. The sweep says otherwise — 34.4% sits beside signedintake's 27.7%, not below it. The two ranges were unrepresentative: one a cross-cutting ledger commit, one containing a file with no indexed symbols. The registry note briefly carried the wrong claim and was rewritten. The harness exists precisely because that intuition is unreliable.

The populated middle is not a counter-example

testgraph's histogram occupies 1, 2 and 3 where mealie's was {0: 38, 23: 2}. That is not evidence against Update 3's bimodality finding: with six journeys the "middle" is only 1–5 wide, so a registry this small cannot exhibit the shape either way. Noted in TECHNICAL.md so it isn't later mistaken for one.

What this is not evidence for

The instrument and the target are the same repo. Selectivity survives that better than recall — harness/selectivity.py checks each commit into its own worktree, builds its own index, and never sees who wrote the registry — but observed_recall from the ledger will be testgraph scoring testgraph against a registry authored while reading testgraph's source.

Treat the ledger's testgraph rows as a liveness test of the selection → outcome join, not as an accuracy claim. The accuracy claims in TECHNICAL.md still come from honeyslate's hand labels and the outside repos in Update 3, and nothing here changes them.

The registry test closes a real gap

Nothing validated any shipped registry. That is exactly the ae128bb finding-8 defect: a trailing comma makes json.load raise, reg.resolve_for_repo swallows the ValueError and returns None, the hook logs NO_REGISTRY and exits 0, and record tells you to draft a file you already have. A typo silently disables the tool for that repo until somebody notices it stopped talking.

Index-free by design — honeyslate's and signedintake's codegraph databases are not present in every checkout, so asserting their entries resolve would fail for reasons unrelated to the registry.

Both guards mutation-tested rather than assumed green (the repo's own standard, after five of seven skill-contract guards once passed against the defect they named):

mutation result
unparseable registry added to journeys/ FAILED (failures=1, errors=1)
second file claiming target testgraph FAILED (failures=2)
restored OK

Operational finding: codegraph cannot find a bare-worktree repo root

This blocked the work entirely and is worth knowing beyond this repo.

In the claudew layout a worktree's .git is a file (gitdir: .../.bare/worktrees/main), not a directory. codegraph 1.5.0's root detection wants a directory, so from any worktree it walks up past the repo. codegraph index <path> then rebuilds whichever ancestor index it finds rather than creating a local one — or refuses at $HOME when there is none.

A 279 MB ~/personal_projects/.codegraph (11,141 files, 55 repos, truncated mid-run with 880,880 unresolved references) had been sitting there answering every query from any un-indexed repo under that root. It had to be removed before testgraph could be indexed at all.

Use codegraph init <path> for a first index, never codegraph index <path>init creates in the directory named, index resolves an ancestor.

Not in this PR

Success criteria 3 and 4 are gated on merge: hooks/install.sh bakes TESTGRAPH_HOME=.../testgraph/main, so the registry has to be on main before the hook can see it — the installer's own note says to install after merging. Once merged: install the hook, push, and confirm a status: OK selection row lands in the ledger.

Verification

243 tests green, was 238.

The results ledger shipped in #57 and held zero rows. Not a bug — an absence of
opportunity. `hooks/install.sh` installs only into repos with an approved
registry, and the only two were honeyslate (last pushed 2026-07-17) and
signedintake (2026-07-19). testgraph is the most actively pushed repo here and
was the one repo the hook did not cover.

`propose --repo .` correctly wrote nothing: 0 route handlers, and it named the
reason from its own blind-spot list (CLI entry points). So the registry is
hand-authored — one journey per CLI surface, 19 entry nodes, all six resolving
with no approval or drift warning.

Selectivity, measured with Update 3's methodology correction applied from the
start (`harness/selectivity.py`, fresh per-commit index, 20 commits of main):
mean 2.07/6 (34.4%) over the 15 commits that touch product code, 65.6% of
journey-runs avoided. Histogram {0:5, 1:8, 2:4, 3:1, 6:2}. All five
zero-selections are exactly the five non-product commits, and every
product-touching commit selected at least one journey.

I got this wrong once mid-task and the correction is worth keeping: two
hand-picked ranges each named 6/6, and I concluded selectivity had collapsed on
a small tightly-coupled codebase. The sweep says otherwise — 34.4% sits beside
signedintake's 27.7%. The two ranges were unrepresentative. The harness exists
because that intuition is unreliable; the registry note carried the wrong claim
briefly and was rewritten.

What this is NOT evidence for: instrument and target are one repo. Selectivity
survives that better than recall, since the harness never sees the registry's
authorship, but observed_recall from the ledger will be testgraph scoring
testgraph. Treat the ledger's testgraph rows as a liveness test of the
selection -> outcome join, not as an accuracy claim. The accuracy claims in
TECHNICAL.md still come from honeyslate's hand labels and the outside repos.

`tests/test_registries.py` closes a gap nothing covered: no test validated any
shipped registry, which is exactly the ae128bb finding-8 defect — an
unparseable registry reads as "no registry", so the hook logs NO_REGISTRY, exits
0, and a typo silently disables the tool until somebody notices it stopped
talking. Index-free by design (honeyslate's and signedintake's codegraph dbs are
not present in every checkout). Both guards mutation-tested: an unparseable file
and a duplicate target each fail the suite.

Also recorded in TECHNICAL.md, because it blocked this work entirely: codegraph
cannot find a bare-worktree repo root. A worktree's `.git` is a file, not a
directory, so codegraph 1.5.0 walks up past the repo — and `codegraph index
<path>` rebuilds whichever ancestor index it finds rather than creating a local
one. A 279 MB stray index at ~/personal_projects had been answering every query
from any un-indexed repo under that root. Use `codegraph init <path>` for a
first index, never `codegraph index <path>`.

243 tests (was 238).
Five review findings. The first falsifies a claim this PR published.

1. `testgraph/db.py` and `testgraph/registry.py` are reachable from NO journey.
   Seeding every node in either file and running the full impacted closure
   intersects the entry map in zero nodes: db.py's closure is 14 nodes and never
   leaves the file, registry.py's is 24, against ledger.py's 257 -> all six
   journeys. So a commit rewriting the whole of db.py — the graph traversal this
   tool IS — answers `journeys to test: NONE` with status OK and
   recall_degraded false.

   The 20-commit window could not see it because no commit in it touches either
   file alone, which makes the "zero silent nothing-affected answers" line in
   Update 5 an artifact of co-changed files. That line is now removed and
   replaced with the measurement that refutes it.

   The cause is not the registry and cannot be fixed there: codegraph records no
   cross-file edges for ALIASED relative imports. Cross-file inbound edge counts
   are exactly 0 for the two modules imported `from . import X as Y`, versus 15
   for integrity, 44 for ledger and 15 for hook, all unaliased. Adding these
   files' symbols as journey entries would manufacture the right answer from a
   false claim about what an entry is, so the registry declares the gap instead.
   A NONE after a db.py or registry.py change means UNKNOWN until the indexer
   links aliased imports.

2. No `spot_checks`, so `integrity.check` received `{}` and the caller-count
   check silently no-opped — the one check of the three that catches the
   2026-07-17 incident integrity.py exists for, and the one `codegraph sync`
   cannot clear. Added `append` (>=5, live 7) and `StampError` (>=4, live 6),
   the two propose nominated, and a test asserting EVERY shipped registry
   carries them.

3. `test_testgraph_registry_is_resolvable_for_this_repo` passed or failed on
   what the checkout directory is CALLED, because `repo_name` derives the target
   from the directory name. It failed in any worktree not literally named
   `testgraph` — including every scratch worktree selectivity.py and accuracy.py
   create, which is exactly where it runs once testgraph is its own measured
   target. The repo path is now synthesized in a temp dir. Verified: the suite
   is green from a directory named `notthesamename`, which previously failed.

4. The `sorted(journeys, key=journey_sort_key)` guard was vacuous — the key
   cannot raise for any str, so it stayed green under any mutation. It now
   asserts the property the renderer depends on: J2 before J10.

5. Known Limitations still said honeyslate was the only approved registry,
   contradicting Update 5 thirty-five lines above and signedintake.json. Both it
   and the USAGE.md FAQ now say three, and name honeyslate as the only one with
   hand labels — so the only one with a recall figure at all.

244 tests (was 243).
Two corrections to the claim made one commit ago.

1. "Cross-file inbound edge counts are exactly 0 for aliased imports" was too
   strong. `select.py` is also aliased (`from . import select as sel`) and has
   one edge, not zero. The claim was literally true of the two modules it named
   but implied a clean dichotomy that does not hold, and a reader checking
   select.py would have found the counter-example before I did.

   Restated as the measured table across all five same-package modules called as
   `module.attr()`: registry.py 38 call sites -> 0 edges, db.py 18 -> 0,
   select.py 6 -> 1, ledger.py 10 -> 44, integrity.py 3 -> 15. The mechanism is
   sharper than "aliased imports" too: codegraph does not resolve calls through a
   module bound under an alias, because the local name no longer matches the
   module name. Same import form, same call form, alias the only difference.

2. Checked whether this reaches honeyslate, since honeyslate is the only repo
   whose numbers are quoted as ACCURACY rather than sizes. It does not.
   honeyslate has 9 aliased imports and every one aliases a symbol
   (`Session as DbSession`), never a module, so the `alias.attr()` form never
   arises. The recall and precision figures in TECHNICAL.md are unaffected, and
   that is now recorded where the limitation is described rather than left for
   the next reader to re-derive.

244 tests.
@inth3shadows
inth3shadows merged commit ad57d1d into main Aug 6, 2026
1 of 2 checks passed
inth3shadows added a commit that referenced this pull request Aug 6, 2026
…#61)

`hooks/install.sh` was committed mode 100644. README.md and USAGE.md both
document running it directly:

    hooks/install.sh                    # every repo with an approved registry
    hooks/install.sh /path/to/repo      # or just one
    hooks/install.sh --uninstall

All three fail with "Permission denied". Found by following the documentation
while installing the hook into testgraph itself after #59 merged — the whole
point of that PR was to close the gap where nothing ever called the selector,
and the documented way to start calling it did not run.

It went unnoticed because every previous invocation was `bash hooks/install.sh`,
which ignores the mode bit entirely, so the file's own tests and every manual run
passed while the documented path did not.

`hooks/pre-push` is deliberately left 100644: it is a template that gets sed'd
into the hook file, and install.sh chmods the OUTPUT. Nothing executes the
template in place.

No behaviour change beyond the mode bit; 251 tests unaffected.
inth3shadows added a commit that referenced this pull request Aug 14, 2026
* measure: trace-derived ground truth against testgraph itself (#12)

honeyslate's Postgres is still unreachable (port :55433 closed, no Docker
under WSL), so this runs the cheaper target Update 4 proposed: testgraph's
own 251-test stdlib suite against its own registry from #59. pytest runs the
existing unittest suite unmodified.

New harness/journey_tests_testgraph.json maps each tests/test_*.py file to a
journey, checked against which module each file exercises UNMOCKED (test_hook
patches select/registry on every path; test_core calls select/export for
real).

81 traced symbols land outside the static footprint across all 6 journeys.
62 are the module-alias blind spot Update 5 already declared, now confirmed
by an independent method. 8 are this map's own file-level over-approximation
noise (test_core.py double-mapped to J2 and J4). 11 are new: ledger.append
has zero real inbound static edges despite unaliased call syntax identical to
ledger.path two lines away, while 7 unrelated .append() list calls elsewhere
get bare-name-matched to it instead — which is why the registry's own
min_caller_edges spot-check passed while pointing at the wrong callers
entirely. A commit that breaks ledger.append currently selects neither J1 nor
J5.

Full writeup in TECHNICAL.md Update 7. Not fixed here — the edge-resolution
defect is in CodeGraph's resolver, a distinct root cause from the
already-documented alias gap, and gets its own issue.

* fix: correct Update 7's traced_only bucket counts, caught by review

Grouped by file totals instead of (journey, symbol) pairs, which hid two
real errors that happened to cancel in the sum: select.py was undercounted
(1 claimed vs 6 actual) and ledger.py was overcounted (11 claimed vs 4
actual, even though the sentence's own enumeration — "append x2, path,
state_dir" — already summed to 4). harness/plugin/tgtrace.py's 2 traced-only
symbols were omitted from every bucket entirely.

Re-derived from the ground_truth --json output grouped by (journey, symbol):
63 alias blind spot (the 62 already known, plus one more instance in
select.py via propose.py's sel._is_test call — same aliased-module pattern,
not previously listed), 12 map-granularity noise (two distinct mechanisms,
not one), 2 the tgtrace.py blind spot the registry's own note already
declares, 4 the ledger.append edge defect (#66). 63+12+2+4=81.

* fix: ledger.path is one line after ledger.append, not two
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant