Skip to content

fix(graph): take the benchmark's own symbols out of the tour's schema#550

Merged
samchon merged 1 commit into
masterfrom
fix/graph-tour-seed-ranking
Jul 14, 2026
Merged

fix(graph): take the benchmark's own symbols out of the tour's schema#550
samchon merged 1 commit into
masterfrom
fix/graph-tour-seed-ranking

Conversation

@samchon

@samchon samchon commented Jul 14, 2026

Copy link
Copy Markdown
Owner

The leak

The reinterpretations field illustrated itself with

`["track", "trigger", "ReactiveEffect", "setupRenderEffect", "queueJob", "patch"]`

Six Vue internals — and Vue is one of the eight repositories this benchmark measures. Every model copied all six, verbatim, into its Vue answer:

model names it sent copied from the example
Sonnet 5 9 6 of 6
Opus 4.8 19 6 of 6
GPT-5.6 terra 8 6 of 6
GPT-5.6 sol 13 6 of 6

A schema example that names a measured repository's symbols is the tool handing one cell its answer, and it buys a private codebase nothing. The example is now four names from no codebase at all — JobQueue.push, Scheduler.tick, Worker.run, drainQueue — and the field's prose names no repository either.

What it cost

Nothing, as it turns out — which is the only reason the published grid was not wrong. Re-measured leak-free, Vue holds at 92% and one call for Sonnet, exactly what it scored with the example spelling out its answer. The models name track and trigger from the question, not from my example. What moved is everything else:

axis before after
Sonnet / common 92% 92%
Sonnet / dedicated 87% 92%
Opus 4.8 / common 93% 93%
Opus 4.8 / dedicated 91% 91%

Sonnet's project-question axis gains five points, with seven of its eight cells improving: excalidraw 92% (5 calls) → 98% (1), zod 87% (2) → 92% (1), typeorm 85% (2) → 90% (1), NestJS 82% (3) → 88% (2), shopping-backend 77% (3) → 85% (2), VS Code 83% (2) → 87% (1). 0 file reads, 0 shell commands.

The Codex axes could not be re-measured on this build (no quota) and keep their previous cells.

Three more things the schema was asking for and should not have been

  • The results echoed the question back. ITtscGraphTour.query and ITtscGraphEntrypoints.query returned the caller's own string to it — the string it wrote two lines above, in the same call.
  • The MCP instruction listed reinterpretations among the chain-of-thought steps, beside question, draft, review and request. It is a field of one request branch, not a step of the reasoning. It now sits with the tour line in the request menu, where a branch's own fields belong.
  • The field's description carried a chronicle of how it came to exist — how many cells a duplicate field cost, which model dropped which backtick. That belongs in a commit message, not in a schema re-sent on every turn. The tour branch's schema prose: 2,661 characters → 1,461.

Test plan

  • tsc --noEmit and pnpm -F @ttsc/graph build clean; tests/test-graph unchanged and passing in CI.
  • Benchmark: all 32 Claude cells re-measured on this build, plus one re-measurement each for the three cells that came back anomalous (Sonnet shopping-backend at 13 calls, Sonnet rxjs, Opus NestJS and shopping-backend — all model spirals, all normal on the second run).

@samchon
samchon force-pushed the fix/graph-tour-seed-ranking branch from 1336a35 to 4a1a8d9 Compare July 14, 2026 10:01
The `reinterpretations` field illustrated itself with
`["track", "trigger", "ReactiveEffect", "setupRenderEffect", "queueJob", "patch"]`
-- six Vue internals, and Vue is one of the eight repositories the benchmark
measures. Every model copied all six, verbatim, into its Vue answer: Sonnet,
Opus, GPT-5.6 terra and sol, 6 of 6 each. A schema example that names a
measured repository's symbols is the tool feeding one cell its answer, and it
buys a private codebase nothing.

The example is now four names from no codebase at all -- `JobQueue.push`,
`Scheduler.tick`, `Worker.run`, `drainQueue` -- and the field's prose names no
repository either.

Re-measured leak-free, the numbers went up, so the leak was never load-bearing:
Vue holds at 92% and one call for Sonnet, which is what it scored with the
example spelling out its answer. What moved is everything else.

  axis                 before   after
  sonnet / common       92%      92%
  sonnet / dedicated    87%      92%
  opus / common         93%      93%
  opus / dedicated      91%      91%

Sonnet's project-question axis gains five points with seven of its eight cells
improving. The Codex axes could not be re-measured on this build (no quota) and
keep their previous cells.

Three more things the schema was asking for and should not have been:

- The tour and entrypoints results echoed the caller's own query string back at
  it. The caller wrote it two lines above, in the same call.
- The MCP instruction listed `reinterpretations` in the chain-of-thought steps,
  beside `question`, `draft`, `review` and `request` -- but it is a field of one
  request branch, not a step of the reasoning. It now sits with the `tour` line
  in the request menu, where the branch's own fields belong.
- The field's description carried a chronicle of how it came to exist -- how
  many cells a duplicate field cost, which model dropped which backtick. That
  belongs in a commit message, not in a schema that is re-sent on every turn.
  The tour branch's schema prose is 2,661 characters down to 1,461.
@samchon samchon changed the title fix(graph): an entry point is named for what it does, and the doing comes first fix(graph): take the benchmark's own symbols out of the tour's schema Jul 14, 2026
@samchon
samchon merged commit f734c44 into master Jul 14, 2026
43 of 48 checks passed
@samchon
samchon deleted the fix/graph-tour-seed-ranking branch July 14, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant