Send claude's haiku alias fully qualified, so claude:haiku runs Haiku - #58
Merged
Conversation
Under `--permission-mode plan` — which orca passes for every ToolSet.ReadOnly/NetworkOnly turn, so every reviewer — the claude CLI serves claude-sonnet-5 for `--model haiku` while still naming haiku in its init line. A `reviewAgent = claude:haiku` pin therefore ran a frontier model at 3x the intended rate. Resolve the bare alias to claude-haiku-4-5 at claude's single `--model` choke point, which the CLI honours in plan mode. Only `haiku` is rewritten: sonnet/opus/fable resolve to their own families in plan mode, and leaving those bare keeps them tracking the latest model in their tier.
adamw
added a commit
that referenced
this pull request
Aug 2, 2026
Cache writes and cache reads sit at opposite ends of base input — a
write bills
*above* it (2× at the one-hour cache TTL, 1.25× at the five-minute one),
a read at
roughly a tenth of it — but orca carried a single cache counter and
`ModelPricing` had no cache-write rate at all. Where the CLI reports no
`total_cost_usd` (`CostTracker.costFor` prefers the reported figure and
only
estimates when it is absent), every written token was priced as if it
had been
read, and the summary hid what is routinely a run's largest single line
item.
### T5.1 — split cache writes from cache reads in `Usage`
`Usage` gains `cacheWriteInputTokens`. It is a *second, disjoint*
sub-portion of
`inputTokens` alongside `cacheReadInputTokens`, so the documented
invariant becomes
`cacheReadInputTokens + cacheWriteInputTokens <= inputTokens`. `Usage.+`
sums the
new axis independently, which keeps it a lawful (commutative,
associative)
addition across a run that mixes backends reporting different subsets of
the
axes.
**Adapters changed** (four, not one):
| backend | change |
|---|---|
| claude | `cache_creation_input_tokens` and `cache_read_input_tokens`
were summed into one number; now each lands on its own axis (the total
input still sums all three wire categories) |
| pi | same fold of `cacheWrite` + `cacheRead`; now split — **and the
input total is fixed**, see below |
| opencode | had `cache.write` on the wire and folded it into
`inputTokens` only; now also reported on its own axis |
| codex | `cache_write_input_tokens` was never declared on the wire
shape; now parsed. codex emits no cost, so the table is its only signal
and the omission silently under-counted |
**pi's input axis was also wrong, independently of the split.** pi
normalises
`usage.input` to the *fresh* prompt only — it forwards Anthropic's
`input_tokens` (which excludes both cache categories) and explicitly
subtracts
cached tokens on providers that include them, then reports `totalTokens`
as the
sum of all four axes. orca was treating `input` as the total, so a
cached pi
turn reported 1k input against a real 61k prompt, and
`Pricing.estimate`'s clamp
then billed no fresh input at all. It now sums the three input axes like
claude
and opencode.
**Adapter deliberately unchanged:** gemini genuinely has no cache-write
counter
on the wire, so it stays read-only with the write axis pinned at zero by
an
explicit assertion.
`Usage` is not persisted to progress logs or run manifests (it only
travels
through `OrcaEvent.TokensUsed`), so there is no on-disk migration.
### Naming: the two counters are `cacheRead` and `cacheWrite`
Review asked what the difference was between `cachedInputTokens` and
`cacheWriteInputTokens`, and why they weren't named symmetrically. They
are the
two directions of the same prompt cache — tokens served **from** it, and
tokens
written **into** it — and the old pair hid that: "cached" and "cache
write" are
not opposites, so nothing in the name told a reader that the first
counted
reads. Renamed for symmetry:
- `Usage.cachedInputTokens` → **`cacheReadInputTokens`**
- `ModelPricing.cachedInputUsdPerMillion` → **`cacheReadUsdPerMillion`**
Read/write is also the vocabulary most of the wire already speaks —
opencode
sends `cache.read` / `cache.write`, pi `cacheRead` / `cacheWrite`, codex
`cache_write_input_tokens`, Anthropic `cache_read_input_tokens`. Only
Anthropic's write side says "creation", which the scaladoc now names as
the
synonym it is, so the wire term stays findable without the field
inheriting its
asymmetry.
A clean rename rather than an alias: orca is 0.x and owes no backwards
compatibility (AGENTS.md "Versioning"). It is purely mechanical — no
arithmetic, numeric literal, or field order changed, and the
captured-turn
regression test below still reproduces the CLI's reported $0.1876374
untouched.
### T5.2 — cache-write rate in `ModelPricing`
`cacheWriteUsdPerMillion` is a new field, and `Pricing.estimate` now
bills three
input categories: fresh input at the base rate, reads at the read rate,
writes
at the write rate, with the fresh remainder being whatever neither
claimed.
**One rate, not a 5m/1h pair.** The tier is a property of the *request*,
chosen
by the CLI: orca never builds a request body, and nothing it does can
move a
write from one tier to the other. Two of the three write-reporting
backends
*can* report the tier — claude's API breaks writes down under
`cache_creation.ephemeral_1h_input_tokens`, and pi exposes
`cacheWrite1h` and
bills short and long writes at different rates in its own cost model —
but
opencode reports a single undifferentiated `cache.write`, so a tiered
`ModelPricing` would need a `Usage` axis one of them can never fill. And
the
intersection is thin where it matters: the estimate only runs when a
backend
omits its own cost figure, and the two backends that can name the tier
are
exactly the two that normally report cost, so a tiered table would spend
two
extra `Usage` axes on a safety net. If that changes, the single rate
becomes a
pair without disturbing anything else.
The shipped Anthropic rows use the **one-hour write tier** (2× base
input),
because that is what the CLI in front of them requests. Measured, not
assumed:
across the live runs every cache-creation token the CLI reported landed
in
`cache_creation.ephemeral_1h_input_tokens` and the five-minute bucket
was
empty, and re-deriving each turn's cost from its own token counts
reproduces
the CLI's reported `total_cost_usd` exactly at those rates. The 5m tier
(1.25×) is pi's default; the table comment names those figures for
anyone
overriding. Where a provider has no per-token write charge, the rate is
the
plain input rate.
**Every row was re-checked against the provider pricing pages** on
2026-08-02,
which is what the `lastUpdated` bump certifies:
- OpenAI's GPT-5.6 family charges 1.25× input for a cache write —
`gpt-5.6-sol`
and `gpt-5.6-luna` were wrong; only the earlier models write for free.
- Stale base rates corrected: `gpt-5.6-terra` (2.00/0.20/12),
`gpt-5.6-luna`
(0.20/0.02/1.20), `gpt-5.4-mini` (0.75/0.075/4.50), and the cache-read
rates
for both Gemini rows (0.125 and 0.03).
- **No row for `claude-haiku-4-5-20251001`.** Turns carrying that id
were
measured billing at Sonnet 5 rates, 3.14× the Haiku row the `-\d{8}`
suffix
fallback bridges them to. The id was not the cause: **plan mode has a
capability floor that upgrades a bare alias below it**, verified against
the
CLI directly — `--model haiku` is answered by
`claude-haiku-4-5-20251001`,
while `--model haiku --permission-mode plan` is answered by
`claude-sonnet-5`. Orca passes plan mode for every ReadOnly turn and
every
reviewer is ReadOnly, so all the measured traffic took the upgrade, and
the
haiku id reaching orca came from the init line, which still announces
the
requested model. The mispricing is therefore an alias-resolution bug
(fixed
separately in #58), not a pricing gap — and a row would have
*overcharged*
genuine Haiku threefold, since the dated id is exactly what un-upgraded
Haiku reports.
- Claude Sonnet 5 keeps its **sticker** rates ($3/$0.30/$6/$15) even
though
Anthropic publishes an introductory $2/$10 through 2026-08-31: the CLI
computes its reported cost at sticker, matching every measured Sonnet 5
turn
exactly and the introductory rates never. A summary mixes reported and
estimated figures, so the table follows the number users reconcile
against.
The table's freshness is enforced, not just documented: a test fails
once
`lastUpdated` is more than 183 days old. Because it fires on the
calendar
rather than on a change, its failure message now names the table and its
file,
says what fixing it means, and states outright that elapsed time is the
trigger — so building an old tag long after its release explains itself
instead
of looking like a regression.
### T5.3 — surface the split in the summary
`formatUsage` rendered one undifferentiated `(40K cached)`. It now
renders
`(40K cache read, 5K cache write)`, dropping either part when zero — so
a
single-counter backend still reads naturally and a cache-heavy run shows
the
write figure that dominates its bill.
### The positional-construction hazard
`Usage` and `ModelPricing` are both public exports and are constructed
positionally in-tree (`Usage.empty`, ~30 sites in `CostTrackerTest`, and
`Pricing.scala`'s own documented example). Both new fields go **last**,
so no
existing positional argument can silently rebind to a different field.
Defaults
then differ deliberately:
- `Usage.cacheWriteInputTokens` **defaults to 0** — that is the
*correct* value
for a backend with no write counter, so the default is a fact rather
than a
placeholder, and it keeps every existing positional call site valid.
- `ModelPricing.cacheWriteUsdPerMillion` has **no default** — there is
no
correct universal value (Anthropic bills writes above input, OpenAI's
older
models at no charge), so a default would silently misprice any table
that
forgot the field, and a wrong rate is invisible in the summary. Without
one,
the compiler flags every table entry instead.
The quiet consequence of the defaulted field is that `Usage`'s
*construction*
keeps compiling while its extractor goes from five fields to six, so a
downstream `case Usage(in, out, cost, cached, reasoning) =>` stops
compiling.
`Usage`'s scaladoc now tells callers to destructure by name, since the
axis
list grows whenever a backend starts reporting a finer breakdown. No
changelog:
orca is 0.x and does not track compatibility.
### Tests
- claude `result` carrying `total_cost_usd` plus both cache counters:
the
reported total is unchanged, and reads and writes render separately.
- The same shape without `total_cost_usd`, against a hand-computed
figure
(100k fresh / 600k read / 300k write = $0.76) — folding writes into
reads
would land at $0.19.
- A backend that over-reports its cache axes past the input total is
clamped to
zero fresh input rather than billed negatively.
- pi and opencode round-trip both counters; pi's fixtures (fresh 1 +
read 3,
fresh 4 + write 6) now sum into the input total.
- No double-counting for a single-number backend (write axis zero ⇒ the
estimate
is exactly what it was before).
- `Usage.+` is commutative and associative across three usages reporting
different axis subsets, with `Usage.empty` as its identity.
Rebased onto master. `sbt scalafmtCheckAll` then `sbt clean compile
test`: all
modules build with no warnings, 1773 tests pass.
**On baseline turns that lacked `total_cost_usd`:** not recoverable —
whether
the measured baseline took the estimate path cannot be reconstructed
from the
transcripts, since the reported and estimated paths leave no
distinguishing
trace in the recorded output. The fix matters for every run where the
field is
absent, which is always the case for codex and gemini (neither emits
cost on
the wire at all).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A settings pin of
reviewAgent = claude:haikuwas running Sonnet 5, at 3x theintended rate. This sends the alias fully qualified so it runs Haiku.
The bug
The claude CLI's plan mode has a model-capability floor. Given a bare alias
below that floor it silently substitutes a stronger model; given a
fully-qualified id it honours the request.
haikuis the only alias belowthe floor.
orca passes
--permission-mode planfor everyToolSet.ReadOnlyandToolSet.NetworkOnlyturn (ClaudeArgs.autoApproveArgs), and every reviewerruns
.withReadOnly— so every reviewer turn is a plan-mode turn.Measured against claude CLI 2.1.220, using orca's own spawn shape
(
--print --input-format stream-json --output-format stream-json --verbose):--permission-mode plan --model haikuclaude-sonnet-5❌--permission-mode plan --model claude-haiku-4-5claude-haiku-4-5-20251001✅--permission-mode plan --model sonnet/opus/fable--model haiku(no plan mode)claude-haiku-4-5-20251001✅5/5 reproductions of the failing row. Isolated by bisecting orca's flags:
--permission-mode planis the trigger;--include-partial-messagesand--session-idare not.The mis-resolution is easy to miss because the CLI's
system.initline stillannounces haiku — only the assistant messages and the result's
modelUsagereport
claude-sonnet-5.The affected path is the settings string only:
reviewAgent = claude:haiku→AgentSpec→RoleAgents→withModel(Model("haiku"))→--model haiku— affectedclaude.haikualready producedclaude-haiku-4-5—never affected
The fix
Resolve the bare
haikualias toclaude-haiku-4-5inClaudeArgs, claude'ssingle
--modelchoke point (ClaudeBackendhas one spawn, and the autonomousand interactive paths share this one arg builder).
AgentConfig.cheapModelandAgent.cheapfunnel through the same point, so awithCheapModel(Model("haiku"))pin is covered too.
DefaultClaudeAgent.Haikunow holds the id once, shared by thehaikuaccessorand the arg builder.
Why only
haikuThe other three aliases demonstrably resolve correctly in plan mode, and a bare
alias earns something real: it tracks the latest model in its tier without a
code change. Pinning
sonnet/opus/fablewould trade that live benefit for amaintenance burden on three ids, to guard a floor move that has not happened.
For
haikuthe tracking benefit is already worthless — the alias does not workunder plan mode at all — so qualifying it costs nothing that currently works.
Mapping every alias would also collide with orca's own vocabulary: the
opusaccessor deliberately means
claude-opus-5[1m](1M context), not plainclaude-opus-5, so a settings-levelopusentry would give the codebase twodifferent meanings of "opus".
Staleness
The rewrite freezes both
claude:haikuandclaude.haikuon 4-5 — there is nolonger a "latest haiku" spelling. That is a real cost, stated in
DefaultClaudeAgent.Haiku's scaladoc, which names itself as the one place tobump when a new haiku ships. It adds no new maintenance point: that constant
already had to be updated on a new generation.
The scaladoc records the CLI version verified against, and — because the init
line lies — which field to re-check it on.
Other backends
Checked all four for the same class of bug; none fixed, none demonstrated.
miniis already thequalified
gpt-5.4-mini). Verified: under--sandbox read-onlythequalified id is echoed back unchanged, and a bare
miniis rejected loudlyrather than silently substituted.
provider/model; every accessor is alreadyqualified. The bare-alias precondition cannot arise.
--approval-mode plan, the closest structural analogue, butflash/proare already qualified ids. Not authenticated in thisenvironment, so untested — unknown, not claimed safe.
making it the strongest structural candidate; but its restricted tier is a
--toolsallowlist, not an approval mode. Not authenticated, so untested.Interaction with #49 (
cost-split-cache-writes) — needs reconciling at mergeNot touched by this PR. #49 adds a
claude-haiku-4-5-20251001pricing rowpriced at Sonnet 5 rates, because that id's traffic really was Sonnet.
I independently confirmed the mechanism behind that row: orca attributes cost
via
ClaudeConversation'sresult.model.orElse(initModel), the result messagecarries no
modelfield, so orca falls back to the init line — whichreports
claude-haiku-4-5-20251001while the turn bills at Sonnet rates.Exactly what #49 observed.
This PR changes that row's justification. Once orca stops sending the bare
alias, nothing it spawns reports
claude-haiku-4-5-20251001any more — aqualified request reports the undated
claude-haiku-4-5and prices on theexisting Haiku row. So on merge that row is either dead or, if it is ever hit
again, denotes genuine Haiku traffic that Sonnet pricing would over-charge 3x.
It should be dropped or re-priced at Haiku rates. #49's own comment anticipates
this ("Revisit when the alias resolves to Haiku again"); the trigger is now
"orca no longer asks for the alias".
Tests
Two, both mutation-checked:
haikualias is sent asclaude-haiku-4-5— fails when the rewrite isremoved
sonnetalias is left bare — fails when the rewrite is generalised toother aliases
Neither sets
ToolSet.ReadOnly: the rewrite is unconditional, and gating onplan mode would imply a conditional that does not exist.
Also verified empirically that
--model Haiku(mixed case) resolves to Haikucorrectly, so the exact-lowercase match is right and case-insensitivity would
be wrong.
sbt scalafmtCheckAllandsbt clean compile testpass with zero warnings.