fix(graph): take the benchmark's own symbols out of the tour's schema#550
Merged
Conversation
samchon
force-pushed
the
fix/graph-tour-seed-ranking
branch
from
July 14, 2026 10:01
1336a35 to
4a1a8d9
Compare
The `reinterpretations` field illustrated itself with `["track", "trigger", "ReactiveEffect", "setupRenderEffect", "queueJob", "patch"]` -- six Vue internals, and Vue is one of the eight repositories the benchmark measures. Every model copied all six, verbatim, into its Vue answer: Sonnet, Opus, GPT-5.6 terra and sol, 6 of 6 each. A schema example that names a measured repository's symbols is the tool feeding one cell its answer, and it buys a private codebase nothing. The example is now four names from no codebase at all -- `JobQueue.push`, `Scheduler.tick`, `Worker.run`, `drainQueue` -- and the field's prose names no repository either. Re-measured leak-free, the numbers went up, so the leak was never load-bearing: Vue holds at 92% and one call for Sonnet, which is what it scored with the example spelling out its answer. What moved is everything else. axis before after sonnet / common 92% 92% sonnet / dedicated 87% 92% opus / common 93% 93% opus / dedicated 91% 91% Sonnet's project-question axis gains five points with seven of its eight cells improving. The Codex axes could not be re-measured on this build (no quota) and keep their previous cells. Three more things the schema was asking for and should not have been: - The tour and entrypoints results echoed the caller's own query string back at it. The caller wrote it two lines above, in the same call. - The MCP instruction listed `reinterpretations` in the chain-of-thought steps, beside `question`, `draft`, `review` and `request` -- but it is a field of one request branch, not a step of the reasoning. It now sits with the `tour` line in the request menu, where the branch's own fields belong. - The field's description carried a chronicle of how it came to exist -- how many cells a duplicate field cost, which model dropped which backtick. That belongs in a commit message, not in a schema that is re-sent on every turn. The tour branch's schema prose is 2,661 characters down to 1,461.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The leak
The
reinterpretationsfield illustrated itself with`["track", "trigger", "ReactiveEffect", "setupRenderEffect", "queueJob", "patch"]`Six Vue internals — and Vue is one of the eight repositories this benchmark measures. Every model copied all six, verbatim, into its Vue answer:
A schema example that names a measured repository's symbols is the tool handing one cell its answer, and it buys a private codebase nothing. The example is now four names from no codebase at all —
JobQueue.push,Scheduler.tick,Worker.run,drainQueue— and the field's prose names no repository either.What it cost
Nothing, as it turns out — which is the only reason the published grid was not wrong. Re-measured leak-free, Vue holds at 92% and one call for Sonnet, exactly what it scored with the example spelling out its answer. The models name
trackandtriggerfrom the question, not from my example. What moved is everything else:Sonnet's project-question axis gains five points, with seven of its eight cells improving: excalidraw 92% (5 calls) → 98% (1), zod 87% (2) → 92% (1), typeorm 85% (2) → 90% (1), NestJS 82% (3) → 88% (2), shopping-backend 77% (3) → 85% (2), VS Code 83% (2) → 87% (1). 0 file reads, 0 shell commands.
The Codex axes could not be re-measured on this build (no quota) and keep their previous cells.
Three more things the schema was asking for and should not have been
ITtscGraphTour.queryandITtscGraphEntrypoints.queryreturned the caller's own string to it — the string it wrote two lines above, in the same call.reinterpretationsamong the chain-of-thought steps, besidequestion,draft,reviewandrequest. It is a field of one request branch, not a step of the reasoning. It now sits with thetourline in the request menu, where a branch's own fields belong.Test plan
tsc --noEmitandpnpm -F @ttsc/graph buildclean;tests/test-graphunchanged and passing in CI.