Skip to content

feat(graph): let the caller name the machinery, and stop asking the question twice#548

Merged
samchon merged 1 commit into
masterfrom
fix/graph-question-verbatim
Jul 14, 2026
Merged

feat(graph): let the caller name the machinery, and stop asking the question twice#548
samchon merged 1 commit into
masterfrom
fix/graph-question-verbatim

Conversation

@samchon

@samchon samchon commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Intent

Two changes to the tour, both measured across all 64 agent cells of the code-graph benchmark.

The tour asked for the question twice. IProps.question and ITtscGraphTour.IRequest.query were the same string, two lines apart, and a schema that asks twice gets it once: GPT-5.6 sent { "type": "tour" } with no query in 31 of 32 cells, the validator rejected it, and every Codex tour paid a full extra round trip. The duplicate also lost what it asked for — question keeps the user's backticks and dotted names, which the mention resolver reads, and a model paraphrasing into query drops them. The tour now ranks against props.question.

The tour landed on the wrong machinery, and the model paid for it. A tour ranks its seeds on words, and a codebase names many things alike: "dependency tracking" matches Vue's devtools hook onRenderTracked as readily as track, the function that records the dependency, and "request" matches NestJS's message listener as readily as its HTTP router. The model notices and says so — "the tour surfaced the microservices listener path, not the HTTP router path" — and then spends two more traces fetching the router by hand.

The tour request now carries reinterpretations: a list of symbol names, never a sentence, naming the machinery the caller expects the answer to be made of. Each name resolves the way a handle does. The ones the graph holds take half the entrypoints; the other half stays with what the graph finds central, untouched by the names. A name the graph does not know — or knows several of — is dropped, so a wrong guess is free, prose costs nothing, and [] is the right answer when the question names no machinery.

Scope

  • packages/graph/src/server/runTour.ts — resolve the named symbols, seat them in half the tour, gate them out when the question names nothing.
  • packages/graph/src/structures/ITtscGraphTour.tsquery removed from the request, reinterpretations added.
  • packages/graph/src/structures/ITtscGraphApplication.ts, packages/graph/README.md — the MCP instruction names the field and its shape.
  • experimental/benchmark/graph/ — the harness rejects a tool arm that never called its tool, and publish.mjs stops dropping the index block.
  • website/ — 64 re-measured cells, 74 regenerated charts, and the design/blog prose.

Results

All 64 cells re-measured on today's clients. 0 file reads, 0 shell commands. Every Codex cell answers in a single call with no validation failures.

axis before after
Sonnet / common 93% 92%
Sonnet / dedicated 83% 87%
Opus 4.8 / common 93% 93%
Opus 4.8 / dedicated 90% 91%
GPT-5.6 terra / common 89% 87%
GPT-5.6 terra / dedicated 85% 90%
GPT-5.6 sol / common 94% 96%
GPT-5.6 sol / dedicated 93% 95%

Sonnet's Vue tour goes from seven calls to one, Opus's NestJS tour from three to one.

Deferred

Whether a tour should ask the caller for symbol names at all is open: the tour is the request you make when you do not yet know them, and a model that guesses names for a repository it has never seen guesses from convention — or, on a famous open-source repository, possibly from memory. The alternative is to fix the seed ranking itself (runtimeEntryScore grants +70 for a verb in the name, which is why a devtools hook with one execution edge outranks the tracker with thirteen). That is being discussed and may replace this.

Test plan

  • pnpm -F @ttsc/graph build and tsc --noEmit clean.
  • tests/test-graph end-to-end MCP case updated for the new tour request; CI runs it.
  • Benchmark: 64 ttsc-graph cells + the three anomalous cells re-measured once each, per the benchmark skill.

The tour ranks its seeds against the query string, so the words in it decide
which symbols are promoted and which are demoted — a 3.2x swing between a symbol
whose name shares a query word and one that shares none. The schema asked for
"the code question being considered", and Sonnet in particular answered that
with its own paraphrase, so the same repository and the same user question
ranked a different tour from run to run.

Say what the field is for: copy the user's words, do not summarize or rewrite
them, and do not append terms you think will help. The ranking is the reason,
so the reason is what the JSDoc gives.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016YXiVdMRPRCTse2qyx9VQH
@samchon
samchon merged commit de9e6aa into master Jul 14, 2026
90 of 95 checks passed
@samchon
samchon deleted the fix/graph-question-verbatim branch July 14, 2026 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant