feat: assign agent models by a measured rule (0.51.0) - #64
Merged
Conversation
Five agents move from haiku to sonnet: tp-run, tp-api-surface-tracker,
tp-dep-health, tp-doc-writer, and tp-incident-timeline — the last was
`inherit`, so it silently took whatever the session ran and could land on
haiku for a production incident.
The old split had no stated basis. Two measurements on identical prompts
through tp-run gave one:
list exported symbols, strict format
haiku 19,020 tok / 9.4s — format exact
sonnet 25,798 tok / 12.6s — added a verdict line the prompt forbade
find a defect in a decision function
haiku 27,916 tok / 89s — narrow type issue
sonnet 30,307 tok / 50s — the functional defect plus the follow-up
Haiku wins shape-bound work outright: cheaper, quicker, more literal about
the format. Sonnet wins once the answer requires deciding what matters,
which is precisely where haiku came back narrower.
So: haiku for output with a predetermined shape (commit message,
orientation map, session briefing, coverage list, commits quoted without
interpretation), sonnet for anything that weighs, ranks or explains.
tp-run is in the second group because it takes whatever no specialist
claimed — its difficulty is unknown at dispatch.
In proportion: the model gap is 8-26%, while dispatching general-purpose
over a specialist costs 3x. The model is the small lever. And since the
gap is small, borderline tasks go to sonnet — a wrong answer costs more
than the model does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five agents move from haiku to sonnet:
tp-run,tp-api-surface-tracker,tp-dep-health,tp-doc-writer, andtp-incident-timeline.That last one was
inherit, meaning it silently took whatever model the session happened to be running — so a production incident investigation could land on haiku.Why
The old 9/15/1 split had no stated basis, so there was no way to place a new agent without guessing. Two measurements on identical prompts through
tp-rungave one:Haiku wins shape-bound work outright: cheaper, quicker, and more literal about the format. Sonnet wins the moment the answer requires deciding what matters — which is exactly where haiku's answer came back narrower.
How
The rule, now written into
docs/agents.md:tp-runsits in the second group because it exists to take work no specialist claimed, so its difficulty is unknown at dispatch time.Risk
Cost goes up slightly for the five reassigned agents. Worth keeping in proportion: the gap between models is 8–26%, while dispatching
general-purposeinstead of a specialist costs 3x. The model is the small lever; the agent is the large one.That also settles borderline cases — since the gap is small, they go to sonnet, because a wrong answer costs more than the model does.
Notes for reviewer
Two measurements are two data points, not a study. They agree with each other and with the direction of the change, but if a haiku agent starts producing thin work in practice, that is better evidence than this table and the line should move again.
The five staying on haiku were checked one by one against the rule rather than left alone by default.