fix(graph): make the tour tell the truth about its own coverage, and publish the grid it earned#405
Merged
Merged
Conversation
… loudest Asked how Excalidraw carries an edit from pointer through scene mutation, history, rendering and collaboration, the tour answered with four renderers and a pointer handler -- two of its five seeds were one render function and its own throttled twin -- and said `next: answer`. Mutation, history and persistence, three of the five layers the question names, were not in it at all, so the model went and found them itself: twenty-one further graph calls and 3.6M tokens on a question the tour claimed to have answered. Two things kept them out. The stemmer only stripped inflection, so the noun in the question never reached the verb in the code: "mutation" and "mutate" share five characters and the prefix rule wants six, so `mutateElement` -- the centre of the layer the question asked about -- matched nothing. And the set cover that is supposed to spread the seeds across the question added a flat bonus for covering a new term, which stopped being decisive once the seed score grew a reach term and a wider fan-out cap: a second renderer, covering nothing new, still outscored the first symbol of an uncovered layer. So the stemmer strips derivational suffixes too, and the cover picks by coverage first and score only within a tier, and a seed that merely restates a chosen one -- same file, same name word for word -- cannot take a slot. The same question now seeds `mutateElement` and `History.perform` alongside the renderer and the pointer handler, in a payload that is slightly smaller than before. Measured on zod, where the shared question is answered through the graph: gpt-5.6-sol spends 75,088 tokens in two calls where it spent 96,666 in three, and its saving against the no-MCP baseline goes 86% -> 89%.
Picking by coverage first and score second burned the village to save one house. On Excalidraw's edit pipeline a stats panel and a "go to collaborator" action took two of the five seeds, each on the strength of one rare word the question happened to contain, and the model abandoned the tour and grepped thirteen files: gpt-5.6-terra spent 862,867 tokens where it had spent 49,844. Sonnet's shared-question corpus fell from 87% saved to 58%, with file reads back on repositories whose top seeds had not even changed -- the ordering had reshuffled the flow candidates underneath them. Coverage is a tiebreak, not a veto. It multiplies the score now, so an unimportant symbol covering two new terms still loses to the centre of the codebase, while two comparable symbols are split by what they add. The flat +120 bonus that came before could not do this: it was tuned against scores that topped out near 150, and the reach term and wider fan-out cap left it deciding nothing. The seeds the fix was for survive it -- Excalidraw's tour still opens on `mutateElement` alongside the renderer, the pointer handler and the collaboration entry -- and the repositories it broke are byte-for-byte back to the seeds they had: TypeORM's four query-builder entries, RxJS's `subscribeToArray` / `Observable.subscribe` / `from`.
A tour question names its stages -- "from the pointer interaction through the scene mutation, the history, the rendering and the collaboration" -- and the entrypoints the tour selects are those stages. The flows say what each stage does downstream. Nothing said what runs *between* two of them, which is the question, so the model went and asked: a lookup for the name, a trace from it, a path to the next stage, one round per stage. Sonnet spent 3 to 12 graph calls on a project-specific question that way, and Claude re-sends the whole context on every turn, so those calls are the token bill. The path between two selected symbols is a graph fact, and runTrace's path mode already resolves it, so the tour resolves it once for the stages that actually reach each other. A path that only retraces a flow already told is dropped, a repository whose stages do not reach each other gets no bridges at all (NestJS), and each pick is the shortest path touching a stage no bridge has touched yet -- without that, all four of Excalidraw's bridges landed on `mutateElement`, saying four times how to reach the mutation and never once what the mutation reaches. Measured on Sonnet, against the same build without them: the project-question corpus goes 69% -> 72% saved and the worst cells lose half their calls (excalidraw 12 -> 7, nestjs 6 -> 4), while the shared-question corpus goes 82% -> 83% with vscode falling 6 calls -> 2.
Three changes went in and none of them took the quadrant they were for. The stemmer and the coverage factor put Excalidraw's mutation and history layers into a tour that had seeded four renderers, and the bridges resolved the paths between the tour's own stages, which halved the calls on the worst cells. Measured on Sonnet: the project-question corpus went 59% -> 72% saved. But the shared-question corpus went 87% -> 83% in the same build, and the quadrant this was all for still belongs to codegraph at 78%. So the trade was 13 points on one family for 4 on the other, and the target was never reached. Two further attempts made it worse, each for a reason the change itself had introduced and I had not predicted: giving the flow nodes their signatures grew the payload 22% and Sonnet answered with *more* calls, not fewer; dropping `answerAnchors` as a repeated coordinate took away what the model was using as its citation list, and Excalidraw went from 7 calls to 19. That is three surprises in three changes, which is the definition of not understanding the thing being changed. The measurements stay (they are what this branch is for); the ranking goes back to master, and the tour's blind spot on question-named sinks -- Vue's `track` is called only from closures, so the ranking sees a function nothing calls -- is written up rather than patched.
…win seeds Excalidraw's project question asks how an edit travels from the pointer through the scene mutation, the history, the rendering and the collaboration. The tour seeded `handlePointerMove` beside `handlePointerMoveInEditMode`, and `renderNewElementScene` beside its own throttled twin -- four slots on two symbols -- and the mutation the question named took none, because "mutation" and `mutateElement` share five characters and the prefix rule wants six. Sonnet spent 22 graph calls building the tour it had asked for. The stemmer now strips derivational suffixes, so the noun in the question lands on the same stem as the verb in the code, and a seed that restates a chosen one -- same file, same name word for word -- cannot take a slot. Both are properties of English and of the graph, not of any repository: no fixture, question or symbol name appears anywhere in the change. The blast radius was computed before the change was measured, by diffing the tour payload of all sixteen (repository x prompt family) cells offline: thirteen are byte-identical and cannot have moved, and three do. On those three, Sonnet's project-question saving goes 4% -> 39% on Excalidraw (22 calls -> 12), 79% -> 85% on zod, and 76% -> 70% on TypeORM; the corpus goes 59% -> 66%. It does not take the quadrant -- codegraph holds it at 78%, by shipping source bodies at 160k tokens a call, which is the trade this tool exists to refuse. Excalidraw's twelve calls are not the tool failing: each one reaches a stage the question named and a five-seed tour could not hold, and none of them re-asks anything the tour already gave.
The tour answered `next: answer` — "the tour covers the question" — on every result it ever returned, including the ones that plainly did not. Excalidraw's project question names the pointer, the mutation, the history, the rendering and the collaboration; the tour has five seed slots, two go to renderers, and the history layer is a mid-level sink no ranking will lift into a seed. It claimed completeness anyway. Sonnet does not take that claim on faith. It went and found the missing stages itself — twelve graph calls, not one of them re-asking anything the tour had given, every one of them reaching a stage the tour had not — and that is the honest reading of a server that lies about its own coverage: the model is right and the payload is wrong. So the tour now says what it left out, when what it left out exists. A term of the question that no seed and no flow touched, and for which the graph holds a symbol that would have belonged among the entrypoints had a slot been free, is a stage the tour owes the reader: `next` becomes `inspect` and names the one request that closes the gap. `answer` is left for the tours that earned it. The bar is the tour's own weakest seed, so there is no threshold to tune per repository, and it is deliberately hard to clear: every question word has some identifier behind it in a large repository, and an `inspect` on every tour would cost a second call to the models that need none — Opus and both Codex tiers answer these questions in one. Checked offline across all sixteen (repository x prompt family) cells before a token was spent: the eight shared questions all still say `answer`, and seven of the eight project questions now say `inspect`. A question about TypeORM also says "TypeORM", which splits into `type` and `orm` and matches a hundred identifiers while saying nothing about which part of the project is being asked about. A term the project's own name contains is dropped now, which moves one more cell's seeds (NestJS: the interceptor context creator takes the slot the HTTP adapter had).
Four rules, each paid for: compute a tour change's blast radius offline over all sixteen cells before measuring one; re-measure every ttsc-graph cell on any logic or text change, because the offline diff says what the server did and not what the model does with it; keep the audit and `next` honest rather than persuasive, since a tour that covers two of five stages and says `answer` is lying and the model that drills after it is right; and treat a surprise as proof the change was not understood, rolling back instead of patching the patch.
Every ttsc-graph cell re-measured on one build -- four models, two prompt families, eight repositories -- because a change to the tour changes what every model sees, and a grid assembled from two builds is not a measurement. | axis | ttsc-graph | codegraph | serena | codebase-memory | | --- | --- | --- | --- | --- | | Sonnet / shared | 81% | 68% | 13% | -8% | | Sonnet / project | 79% | 78% | -11% | 15% | | Opus / shared | 93% | 65% | 26% | 25% | | Opus / project | 88% | 71% | 28% | 26% | | GPT-5.6 terra / shared | 88% | 22% | -23% | -12% | | GPT-5.6 terra / project | 83% | 31% | -26% | 0% | | GPT-5.6 sol / shared | 96% | 52% | -5% | 23% | | GPT-5.6 sol / project | 89% | 42% | -26% | 3% | Eight quadrants, eight wins. The one that had never been won -- Sonnet on the project questions, where codegraph has always held 78% -- goes 59% -> 79%, and not by making the answer cheaper: by making the server stop claiming a completeness it did not have. Sonnet drilled twelve to twenty-two calls because the tour said `answer` on a payload that carried two of the five stages the question named. It says `inspect` there now, and names the one request that closes the gap, and the same corpus lands in 3.4 calls.
samchon
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The result
Eight quadrants, eight wins — the first time
@ttsc/graphhas taken all of them.(Tokens saved against the no-MCP baseline; a negative number means the tool cost more than running with no tool at all.)
Sonnet on the project questions is the quadrant that had never been won — codegraph has held it at 78% from the first measurement. It goes 59% → 79%, and not by making the answer cheaper.
Why it was lost, and what actually fixed it
The tour returned
next: answer— "the tour covers the question" — on every result it had ever returned, including the ones that plainly did not. Excalidraw's question names the pointer, the mutation, the history, the rendering and the collaboration; the tour has five seed slots, two go to renderers, and the history layer is a mid-level sink no ranking will lift into a seed. It claimed completeness anyway.Sonnet does not take that claim on faith. It went and found the missing stages itself — twelve to twenty-two graph calls, not one of them re-asking anything the tour had given, every one reaching a stage the tour had not. Read the trace and the model is right and the payload is wrong: a server that lies about its own coverage teaches the model to distrust it.
So the tour now says what it left out, when what it left out exists:
is a stage the tour owes the reader.
nextbecomesinspectand names the one request that closes the gap;answeris left for the tours that earned it. The bar is the tour's own weakest seed, so there is nothing to tune per repository — and it is deliberately hard to clear, because every question word has some identifier behind it in a large repository and aninspecton every tour would cost a second call to the models that need none. Checked offline across all sixteen (repo × family) cells before a token was spent: all eight shared questions still sayanswer; seven of the eight project questions sayinspect. The same corpus that took 12–22 calls now takes 3.4.Two smaller fixes ride with it, both found the same way — by reading a wrong tour, not by chasing a number:
mutateElement(five shared characters, and the prefix rule wants six). It strips derivational suffixes now.handlePointerMovebesidehandlePointerMoveInEditMode,renderNewElementScenebeside its own throttled twin — four slots on two symbols. A seed that restates a chosen one (same file, same name word for word) cannot take a slot.typeandormand matches a hundred identifiers while naming no stage. A term the project's own name contains is dropped.No repository, question or symbol name appears anywhere in the change; every rule is a property of English morphology or of graph shape.
What was tried and rolled back
Three changes fixed the cell they targeted and cost more elsewhere than they gained, and each was reverted the moment it surprised me:
answerAnchorsas a "repeated coordinate" took away what the model was using as its citation list — Excalidraw went from 7 calls to 19.The rules those cost are written down in
.codex/skills/benchmark/SKILL.md: compute a change's blast radius offline across all sixteen cells before measuring one, re-measure everyttsc-graphcell on any logic or text change, keep the audit andnexthonest rather than persuasive, and treat a surprise as proof the change was not understood.Measurement
Every
ttsc-graphcell was re-measured on this one build — 4 models × 2 families × 8 repositories = 64 cells — because a change to the tour changes what every model sees, and a grid assembled from two builds is not a measurement. The baseline and comparator arms never touch this server and stand as measured.🤖 Generated with Claude Code
https://claude.ai/code/session_016YXiVdMRPRCTse2qyx9VQH