Skip to content

feat(pr-workflow): add lane-graphs — a router contract and three executable lanes - #107

Draft
MajorLift wants to merge 1 commit into
mainfrom
jongsun/add/lane-graphs
Draft

feat(pr-workflow): add lane-graphs — a router contract and three executable lanes#107
MajorLift wants to merge 1 commit into
mainfrom
jongsun/add/lane-graphs

Conversation

@MajorLift

@MajorLift MajorLift commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Adds lane-graphs: a router contract plus three lanes converted from prose to executable graphs.

Why

Today's evidence lanes are prose. A reader follows them by hand, and the mechanical parts get got wrong — repeatedly, in ways that produce a clean artifact:

failure what it looked like
Measured $SHA^..$SHA instead of the pull request's range a twenty-sixth of a change, same runner, same green run, same artifact shape
Copied a probe where its imports did not resolve the suite failed to load; the run reported success and measured nothing
A mutation altered in transit by awk -v escape processing a different test failed than the one targeted, the full suite ran, every guard passed, and the run reported a falsification for a line it never wrote
Published to a merged pull request 22 of 27 published runs, median 22 days after the merge, gate-clean every time

None needs judgement to check, and each is one node in a graph.

The split

Routing is judgement, execution is not. Choosing which measurement answers a claim requires reading the change. Computing a merge-base, verifying a probe loaded, reading a mutation back off disk — none of that requires reading anything.

Five nodes per graph, and a graph missing any is a script:

node job
preconditions the range is the PR's, the target is open, the tree is at a pinned SHA
baseline run it unmutated; fail if the output is not the expected shape
treatment apply one change, read back what was applied
prediction the named observation the treatment should produce
capture the tool writes the artifact; the run URL is the citation

Reading the treatment back, and naming the prediction, exist because they were missing when something published anyway.

The router contract, and the two clauses that matter

No-match is a first-class result. A router over enough lanes always finds a best score. That score means nothing if nothing fit, and a graph run on a claim it does not measure produces a clean green answering a question nobody asked.

An unrouted read runs regardless of what matched. The catalog is a list of questions someone already thought of, so anything outside it is invisible to routing by construction. In the trial that produced this skill, the most valuable finding came from reading a mechanism rather than executing a lane — any router would have matched that change to its signature lane, run it, gone green, and stopped.

Plus: the route is published with the result, so a reader who disagrees with it can say so; and each graph states its own blind spot, because a green that does not say what it declined to measure reads as broader than it is.

Three graphs

assets/*.graph.json — declarative, executed by evidence's existing run workflow and runners:

graph answers
mutation-power does the suite notice the mechanism going away
base-branch-proof is this test connected to the reported bug
render-delta did the work actually decrease

Each carries a does_not_cover list in its own output.

When a lane should not get a graph

Graphs cost maintenance and drift silently — a defect can sit in one for weeks while every run looks correct. Build one when the measurement runs often, its preconditions have been got wrong by hand, and its result is checkable without interpretation. Leave prose where the interesting part is the reading: policy diffs, retention reviews, supply-chain disposition.

The downside, stated plainly

A deterministic graph makes wrongness consistent. A flaky procedure fails visibly; a deterministic one fails identically forever and reads as evidence. Two large graphs reviewed during this work were better engineered than anything here — reproducible, checksummed, hundreds of nodes — and their assertions keyed on a per-case sentinel rather than on absence, so a regression leaking a different value passed every run. This skill does not solve that. It asks each graph to publish what it did not measure, which is the smallest honest mitigation.

Depends on

evidence (#84) for the runners and the run workflow, and falsifiers-first / instrument-check / coverage-partition (#106) — four ## Related links resolve on their merge.

Test plan

  • node .github/scripts/lint-skill-entry.mjs — 0 errors
  • All three graph files parse as JSON
  • Reviewer check: is the five-node shape right, or is a node missing that would have caught something

This branch alone does not work

The installer resolves one source directory. It does not follow ## Related as a dependency edge, does not fetch other refs, and reports a dangling reference as a warning rather than an error — because forward references across open pull requests are expected. So a checkout of this branch installs only the skills on it, with every cross-branch reference dead.

To get a tree that actually runs:

git checkout -b trial/combined origin/main
git merge origin/jongsun/ci/ship-hooks-and-trigger-check \
          origin/jongsun/add/pr-validate-falsifying-test \
          origin/jongsun/add/audit-skills \
          origin/jongsun/add/lane-graphs
node bin/metamask-skills.mjs list --domain pr-workflow --maturity experimental

Merges clean onto main. Yields all eleven pr-workflow skills plus evidence, and lints at 0 errors. Three ## Related names still dangle — react-render-delta, race-condition-repro, lavamoat-policy — because those engines ship in #43, #97 and #83.

Worth knowing that the combination is also where defects surface: an error caught only with #99 and this set together — a description advertising a command the installer does not emit — passes on every branch individually.


Draft: these files are pending removal

The three assets/*.graph.json files on this branch have moved to a standalone runners repository (v0.1.0). That repository is the source; the copies here are temporary. They remain only so this branch can be run and tested on its own while the extracted repository is reviewed.

Once that review lands, they come out and the skill references the runners by pinned tag.

Why the interim needs watching. Two copies of these files is the arrangement that already failed: CI executed the runners from a branch by ref, four fixes landed on the copy it was reading, and none reached the copy under review — including a mutation runner that rewrote the line it was asked to apply, reported the full suite as having run, and published a falsification for a mechanism it never touched. That fix is cited in this description and was, until it was reconciled, still live in the code this branch ships.

So while both copies exist:

git clone <runners-repo>
evidence-runners/tools/check-duplicates.sh <this checkout>

The graphs are not in that comparison yet — they were copied after it was written. A copy made without adding it to the check is a copy nobody is watching, which is the failure in miniature.

What is still open, and it is layout rather than content. A graph reads as documentation and executes as configuration, so whether it belongs to the skill or the runners has not been settled. The router contract and the five-node shape do not depend on that call and are reviewable now.

…cedure

Determinism belongs on the envelope and not on the inquiry. Computing a
merge-base, checking a probe loaded, verifying a mutation landed where it was
aimed — none of that needs judgement, and all of it has been got wrong by hand.
Which measurement answers a claim still does.

Five nodes per graph, two of which exist only because they were missing when
something published anyway: reading the treatment back off disk, and naming the
observation it should produce. A run whose mutation silently changed shape, or
that went red somewhere other than where it aimed, satisfied every other check.

The router contract carries the part that keeps routing honest. A router over
enough lanes always finds a best score, so no-match has to be a first-class
result rather than a fallback into the nearest lane, and an unrouted read of the
mechanism runs regardless of what matched — the catalog is a list of questions
someone already thought of, and the findings worth having sit outside it.

Three lanes converted from prose: mutation power, base-against-branch, and render
delta. Each states its own blind spot in its output, because a green result that
does not say what it declined to measure reads as broader than it is.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant