feat(graph): let the caller name the machinery, and stop asking the question twice#548
Merged
Merged
Conversation
The tour ranks its seeds against the query string, so the words in it decide which symbols are promoted and which are demoted — a 3.2x swing between a symbol whose name shares a query word and one that shares none. The schema asked for "the code question being considered", and Sonnet in particular answered that with its own paraphrase, so the same repository and the same user question ranked a different tour from run to run. Say what the field is for: copy the user's words, do not summarize or rewrite them, and do not append terms you think will help. The ranking is the reason, so the reason is what the JSDoc gives. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016YXiVdMRPRCTse2qyx9VQH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Two changes to the tour, both measured across all 64 agent cells of the code-graph benchmark.
The tour asked for the question twice.
IProps.questionandITtscGraphTour.IRequest.querywere the same string, two lines apart, and a schema that asks twice gets it once: GPT-5.6 sent{ "type": "tour" }with no query in 31 of 32 cells, the validator rejected it, and every Codex tour paid a full extra round trip. The duplicate also lost what it asked for —questionkeeps the user's backticks and dotted names, which the mention resolver reads, and a model paraphrasing intoquerydrops them. The tour now ranks againstprops.question.The tour landed on the wrong machinery, and the model paid for it. A tour ranks its seeds on words, and a codebase names many things alike: "dependency tracking" matches Vue's devtools hook
onRenderTrackedas readily astrack, the function that records the dependency, and "request" matches NestJS's message listener as readily as its HTTP router. The model notices and says so — "the tour surfaced the microservices listener path, not the HTTP router path" — and then spends two more traces fetching the router by hand.The tour request now carries
reinterpretations: a list of symbol names, never a sentence, naming the machinery the caller expects the answer to be made of. Each name resolves the way a handle does. The ones the graph holds take half the entrypoints; the other half stays with what the graph finds central, untouched by the names. A name the graph does not know — or knows several of — is dropped, so a wrong guess is free, prose costs nothing, and[]is the right answer when the question names no machinery.Scope
packages/graph/src/server/runTour.ts— resolve the named symbols, seat them in half the tour, gate them out when the question names nothing.packages/graph/src/structures/ITtscGraphTour.ts—queryremoved from the request,reinterpretationsadded.packages/graph/src/structures/ITtscGraphApplication.ts,packages/graph/README.md— the MCP instruction names the field and its shape.experimental/benchmark/graph/— the harness rejects a tool arm that never called its tool, andpublish.mjsstops dropping the index block.website/— 64 re-measured cells, 74 regenerated charts, and the design/blog prose.Results
All 64 cells re-measured on today's clients. 0 file reads, 0 shell commands. Every Codex cell answers in a single call with no validation failures.
Sonnet's Vue tour goes from seven calls to one, Opus's NestJS tour from three to one.
Deferred
Whether a tour should ask the caller for symbol names at all is open: the tour is the request you make when you do not yet know them, and a model that guesses names for a repository it has never seen guesses from convention — or, on a famous open-source repository, possibly from memory. The alternative is to fix the seed ranking itself (
runtimeEntryScoregrants +70 for a verb in the name, which is why a devtools hook with one execution edge outranks the tracker with thirteen). That is being discussed and may replace this.Test plan
pnpm -F @ttsc/graph buildandtsc --noEmitclean.tests/test-graphend-to-end MCP case updated for the new tour request; CI runs it.