Seeded dual-routing model pins never refresh, and the divergence is invisible

[pacphi]

Summary

Once ak auto-seeds the per-activity routing policy, those model IDs are frozen forever. When MODEL_CATALOG / DEFAULT_ROUTES later move, resolveRoutes keeps preferring the persisted values — so a machine seeded before the bump keeps routing to the older model, and no ak surface reports the divergence.

This repo shipped exactly that bump three releases ago (418e66f, claude-opus-5 as the reasoning default, v4.0.0-alpha.22), and a machine tracking main still routes six activities to claude-opus-4-8.

The defect is the invisibility, not the older model. Measured end-to-end (see below), the diverged pin is cheaper and faster on routine work and the newer default wins at the hard end — so the user is being denied a decision that genuinely goes both ways, and any fix that frames divergence as "stale, upgrade" would be worse than the status quo.

Observed

src/lib/routing.mjs:140-153 (current main):

architecture:        R('claude', 'claude-opus-5'),
design:              R('claude', 'claude-opus-5'),
implementation:      R('codex',  'gpt-5.4', [{ host: 'claude', model: 'claude-opus-5' }]),
testing:             R('codex',  'gpt-5.4', [{ host: 'claude', model: 'claude-opus-5' }]),
'security-analysis': R('claude', 'claude-opus-5'),
debugging:           R('claude', 'claude-opus-5'),

~/.config/agentic-kit/kit.json on the same machine, same checkout:

"architecture":      { "host": "claude", "model": "claude-opus-4-8", "source": "seeded" },
"design":            { "host": "claude", "model": "claude-opus-4-8", "source": "seeded" },
"implementation":    { "host": "codex", "model": "gpt-5.4",
                       "escalate": [{ "host": "claude", "model": "claude-opus-4-8" }],
                       "source": "seeded" },
"security-analysis": { "host": "claude", "model": "claude-opus-4-8", "source": "seeded" },
"debugging":         { "host": "claude", "model": "claude-opus-4-8", "source": "seeded" }

ak x provider status renders these as seeded with no indication they are stale, and ak status's providers row reads ok. The drift has propagated downstream: .agentic-qe/llm-config.json agentOverrides pins qe-security-reviewer to claude-opus-4-8, and the aqe fallback chain's primary rung is claude-opus-4-8 while AQE_FALLBACK_CODEX_SUGGESTION (src/lib/providers.mjs:219) now suggests claude-opus-5.

What the divergence actually costs — and why "just refresh to newest" is wrong

The catalog's annotations (src/lib/routing.mjs:46,50) describe claude-opus-5 as "~2× Opus 4.8 at the same price" and claude-opus-4-8 as "prior Opus generation — same price as opus-5, kept for pinned configs". Read naively that makes the pin a pure loss — same cost, less capability.

Measured end-to-end, it is not. From pacphi/retort versions-blog.md, same task, every arm scoring an identical 1.00:

Python bookshop	n	pass	turns	seconds	cost
Opus 4.8	6	1.00	17.3	122	$0.50
Opus 5	1	1.00	36.0	392	$1.84
Fable 5	3	1.00	10.7	96	$0.76

Confirmed at n=9 across nine further languages: Opus 5 59.8 turns / 712 s / $3.76 at 1.00, against Opus 4.8's 32.2 turns / 375 s / $1.07 at 0.95.

Per-token price is at parity; per-task cost is not, because Opus 5 takes 2–3.4× the agentic turns and cache reads scale roughly with the square of turn count (its Python run generated 33 K tokens and read back 3.28 M). The annotation is a price-per-token statement being read — by users, and by the routing defaults — as a cost-per-task statement.

For this repo's own configuration the dollar figures don't transfer, and the finding gets sharper. Routes here run through claude-code oauth, so marginal cost is $0; what the extra turns consume is wall-clock and subscription quota. 392 s vs 122 s for an identical result is the tax actually being paid, and no ak surface exposes it.

So the direction of the divergence is activity-dependent, not uniformly bad:

• architecture, design, security-analysis, debugging — plausibly the hard end, where the same source notes Opus 5's extra iteration "converts failures into passes" (it is the only model clearing the hard task in all 13 languages). The newer default is defensible here.
• Escalation rungs on implementation / testing — escalation fires because the first attempt failed, i.e. the task proved hard. Also defensible.
• Anything routine — the newer default is measurably worse on every axis that isn't per-token price.

This does not weaken the case for the issue; it changes what the fix must be. Detection must report a divergence, neutrally, with the trade visible — never "you are behind, upgrade."

Root cause

Two mechanisms combine.

1. Seeding is once-only, and resolveRoutes cannot distinguish "machine-chosen, now stale" from "user pinned deliberately."

seedDualRoutingIfDualHost (src/lib/providers.mjs:350-359) bails whenever a policy exists:

const existing = p.dualRouting ?? {};
if (Object.keys(existing).length > 0) return { seeded: false, count: Object.keys(existing).length };

resolveRoutes (src/lib/routing.mjs:184-199) then overlays the persisted entry onto the default with the persisted value winning:

out[act] = {
  host: p.host ?? def.host,
  model: p.model ?? def.model,
  ...
};

The source field already carries the distinction — 'seeded' (ak chose it) vs 'user' (a deliberate pin) — and routingSummary counts both. But no code path acts on it. Nothing re-seeds a seeded entry when the default moves, and nothing flags the divergence. The provenance is recorded and then never used for the one job it is uniquely suited for.

2. aqeFallback has no provenance at all.

"aqeFallback": [
  { "provider": "claude-code", "models": ["claude-opus-4-8"] },
  { "provider": "openai", "models": ["gpt-5.6"] }
]

No source marker. Whether the user typed this or pressed enter on the suggested default is unrecoverable, so even a provenance-aware refresh could not safely touch it. AQE_FALLBACK_CODEX_SUGGESTION has since moved to claude-opus-5 with no path to reconcile.

MODEL_CATALOG_VERIFIED = '2026-07-24' exists and is surfaced in help text — so the catalog already knows it has a freshness date. Persisted policies just don't compare against it.

Impact

• Six activities silently run a model the project no longer routes them to, with no way for the user to discover this short of diffing kit.json against routing.mjs by hand. Whether that is better or worse per activity is exactly the judgment the user is being denied.
• The divergence widens with every catalog bump. Nothing self-corrects, and ak sync's idempotence guarantee actively preserves it.
• ak status reporting ok while the routing policy diverges from the shipped defaults is the same class of incorrect health signal as the credential gap in the companion issue — a green row that means "matches what's on disk", not "matches what ak currently recommends".
• Because divergence is invisible, so is its cost. On a subscription-billed setup the newer default's 2–3.4× turn count is spent from quota and latency budget with no signal at all — and on the diverged machine, the older pin is quietly the cheaper one on routine work.

Proposal

Two mechanisms, deliberately split by risk.

1. Detect and report, neutrally (safe, do first). Add a pure divergedRoutes(policy) to routing.mjs returning every activity where source === 'seeded' and model !== DEFAULT_ROUTES[act].model. Surface it:

• ak status → info (not warn) providers row: routing: 6 seeded routes diverge from current defaults (claude-opus-4-8 vs claude-opus-5) — ak x provider diff
• ak x provider status → mark the row seeded ≠ default with the current default inline

Deliberately info, not warn, and deliberately "diverges from" rather than "stale" or "superseded". Per the measurements above the diverged pin is sometimes the better choice; a warning would push users to spend 2–3.4× the turns to clear a lint.

2. Offer an explicit, informed refresh (opt-in). ak x provider refresh (or pick --refresh-seeded) that re-seeds only source === 'seeded' entries from current DEFAULT_ROUTES, leaves every source === 'user' entry untouched, prints a per-activity before/after diff, and re-projects into .agentic-qe/llm-config.json. Refresh should be per-activity selectable, not all-or-nothing — the right answer genuinely differs across the six.

3. Fix the catalog annotations — LANDABLE STANDALONE, no dependency on items 1/2/4.

The refresh diff and pick's model prompt should carry each candidate's cost-per-task characteristic, not just its ID. The existing note field is the right vehicle — but the notes currently describe per-token price in language that reads as per-task cost, which is exactly the misreading that produced the original (wrong) framing of this issue.

Every entry in MODEL_CATALOG (src/lib/routing.mjs:44-58) states price or abstract capability; none states work-per-task. The three misleading ones:

Model	Current note	Problem
claude-opus-5	"~2× Opus 4.8 at the same price"	true per token; ~3.7× per task on routine work
claude-opus-4-8	"same price as opus-5, kept for pinned configs"	reads as "no reason to keep it" — it is the cheaper arm on routine work
claude-sonnet-5	"near-Opus at lower cost"	same conflation, and unmeasured

Proposed replacements for the first two:

• claude-opus-5 → "new top Opus — same per-token price as 4.8, but ~2–3× the agentic turns on routine work; earns it at the hard end"
• claude-opus-4-8 → "prior Opus generation — same per-token price, roughly half the turns on routine work"

claude-sonnet-5's claim should either be grounded against a measurement or reworded to drop the cost comparison.

This matters beyond the refresh diff: formatModelHelp() (src/lib/routing.mjs:86-97) renders these notes into ak x provider pick --help, the interactive prompt, and the rendered table in docs/PROVIDERS.md:188. Anyone choosing a model today is choosing on a price axis while paying on a turns axis.

Explicitly not proposed: auto-refreshing during ak sync. sync is documented as idempotent reapplication of persisted choice; silently changing which model the user's work runs on breaks that contract. The retort data turns this from a contract argument into a correctness one — an auto-refresh would have tripled turn count and wall-clock on six activities, unannounced, in exchange for nothing on routine work.

4. Stamp aqeFallback with provenance. Record source: 'suggested' | 'user' per entry when written by pick, so the same stale-detection can extend to the chain. Treat a legacy unstamped chain as 'user' (never auto-touch it) and let refresh offer it interactively.

5. Guard against recurrence. A test asserting that every model ID in DEFAULT_ROUTES (including escalate rungs) exists in MODEL_CATALOG would have caught neither of these — the diverged value is in persisted state, not source. The durable fix is a checklist item on catalog bumps: when MODEL_CATALOG_VERIFIED moves, divergence detection must be exercised against a fixture policy seeded from the previous catalog.

Acceptance criteria

• divergedRoutes(policy) is pure, unit-tested, and returns [] for a policy seeded from current defaults.
• A source: 'user' entry pinned to an older model is never reported (deliberate pins are respected).
• ak status emits an info — not warn — providers row listing diverged seeded routes with the current default alongside; a test asserts the severity, since warn would misrepresent the trade.
• Neither ak status nor ak x provider status uses "stale", "outdated", or "superseded" for a diverged route.
• ak x provider refresh re-seeds only seeded entries, supports per-activity selection, prints a diff, and re-projects agentOverrides; a test asserts user entries survive.
• The refresh diff surfaces each candidate model's cost-per-task characteristic, not only its ID.
• (item 3, standalone) No MODEL_CATALOG note states a price or cost comparison without qualifying it as per-token; a test asserts no note matches /(same|lower|higher) (price|cost)/ unqualified.
• (item 3, standalone) formatModelHelp() output carries a one-line header that per-token price ≠ per-task cost, and docs/PROVIDERS.md is regenerated to match.
• ak sync does not auto-refresh (asserted by test).
• New aqeFallback entries carry source; legacy entries default to 'user'.
• Reproduction above (policy seeded pre-alpha.22, checkout on main) reports 6 diverged routes.

Notes

The repo source itself is clean — a grep for claude-opus-4-8 across src/, docs/, and tests/ turns up only legitimate uses: the catalog's prior-tier entry (src/lib/routing.mjs:50), its rendered row in docs/PROVIDERS.md:188, and deliberate test fixtures (tests/kit/routing.test.mjs:64-66,125-137 use it precisely as the user-pinned value). The alpha.22 bump was complete. The drift lives entirely in persisted machine state, which is what makes it invisible to CI and to anyone reading the repo — and why detection has to run against kit.json, not the source tree.

Related: #36 (multi-LLM provider UX). Companion to the aqe fallback-chain credential issue filed alongside this one.

[pacphi]

Companion: #54 — the credential-viability gap in the aqe fallback chain. Both surface through the same ak status providers row and the same ak x provider status table; the detection helpers should land side by side.

[pacphi]

Correction — the original framing of this issue was wrong, and the body has been revised.

As filed, this issue claimed the diverged pin was "strictly worse: same cost, roughly half the capability" and that the fix was to refresh forward. That was derived from the catalog annotation (~2× Opus 4.8 at the same price) — which is a price-per-token statement I read as a cost-per-task statement.

Measured end-to-end in pacphi/retort versions-blog.md, on tasks where every arm scores an identical 1.00:

Python bookshop	n	pass	turns	seconds	cost
Opus 4.8	6	1.00	17.3	122	$0.50
Opus 5	1	1.00	36.0	392	$1.84
Fable 5	3	1.00	10.7	96	$0.76

Confirmed at n=9 across nine more languages (Opus 5: 59.8 turns / 712 s / $3.76 @ 1.00; Opus 4.8: 32.2 turns / 375 s / $1.07 @ 0.95). Mechanism: turn count is what varies, and cache reads scale ~n² in turns.

Three consequences for this issue:

1. The word "stale" is wrong and has been removed throughout. On routine work the diverged pin is the better choice; on hard work the newer default is. The detection helper is now divergedRoutes, the ak status row is info rather than warn, and an acceptance criterion now forbids "stale/outdated/superseded" phrasing — a warn row would push users to spend 2–3.4× the turns to clear a lint.

2. The "no auto-refresh in ak sync" position is upgraded from a contract argument to a correctness one. An auto-refresh would have tripled turn count and wall-clock across six activities, unannounced, buying nothing on routine work.

3. On this repo's own config the dollar column doesn't apply, which sharpens it. Routes run through claude-code oauth at $0 marginal, so the extra turns are spent from wall-clock and subscription quota instead — 392 s vs 122 s for an identical result, with no signal anywhere in ak.

Refresh is now specified as per-activity selectable rather than all-or-nothing, and the diff is required to surface each candidate's cost-per-task characteristic rather than just its ID.

Unchanged: the underlying defect (once-only seeding + resolveRoutes preferring persisted values + source provenance recorded but never acted on), the reproduction, and the aqeFallback-has-no-provenance sub-finding.

#54 is unaffected — credential viability is orthogonal to model economics.

[pacphi]

Sharpened proposal item 3 (catalog annotations) into a concrete change and marked it landable standalone — it does not depend on divergence detection or the refresh command.

Named the three misleading MODEL_CATALOG notes with proposed replacement text, and noted the blast radius: formatModelHelp() renders these into pick --help, the interactive prompt, and the docs/PROVIDERS.md table, so today's users choose on a price axis while paying on a turns axis. Added two acceptance criteria scoped to that item, including a regex test so an unqualified price claim can't reappear.

Rationale: this exact conflation produced the original wrong framing of this issue, so it is worth fixing whether or not the rest lands.