Skip to content

v1.108.311 — The intuition inverts once the block is cached

Choose a tag to compare

@jgravelle jgravelle released this 30 Aug 23:45
· 1 commit to main since this release

A narrowing that cannot repay its own cache invalidation is now refused.

set_tool_tier("standard") and the shipped model_tier_map both offered a mid-session switch that costs more than it saves, for the whole life of any session anyone runs.

tools is serialised ahead of system and messages, so changing the published tool list invalidates the cached prefix — the schema block and every turn accumulated behind it — and the new block must be cache-written before it reads cheaply again. Measured on the live catalog (benchmarks/tier_switch/, regenerable, artifact committed):

switch one-time saved/req break-even
full → core 8,530 2,012 4 reqs
full → standard 31,416 181 174 reqs

With the history the switch also invalidates, full → standard needs 864 requests at 100k of accumulated context.

Why it was invisible: uncached, the same switch saves 1,810 tokens on every request at no one-time cost and pays back immediately. It is wrong only because the block is cachedbenchmarks/codex_surface/ measured 86% of baseline input cached. "Fewer tokens is better" holds right up until the block is stable, which is exactly when it stops holding. That is how a surface built to save tokens shipped a control that spends them.

A widening is never refused. Escalating to a larger surface after a capability-gated failure buys a capability, and trading a correct answer for a cheap one is the worse error. Only a narrowing is judged, because only a narrowing claims to save. standard also remains a perfectly good startup tool_profile — there is no switch to pay for at startup, and the refusal names that route.

Existing installs keep their own model_tier_map on config --upgrade; what was a silent pessimization is now a refusal that explains itself and reports its break-even.

Three defects fell out of building it

  • The first pricing helper filtered the raw catalog by the tier bundle and was wrong by three tools in every tier — it kept the hidden Counter front door and dropped the force-included tier controls, pricing a surface no client receives. _build_tools_list takes a profile_override now, so the price comes from the function list_tools uses rather than a second copy of the visibility rules.
  • The refusal's explanation was written to _meta, which meta_fields: []the default — strips. Most users would have received a bare verdict with the cause removed by a display preference nobody would connect to it. It is reason, in the body. Caught by the test, not by review.
  • The map ships twice, and the first ratchet read DEFAULTS alone — passing while the config template still routed claude-sonnet and gpt-4o at standard.

Two older tests were encoding the defect and were rewritten rather than fixed back.

tests/test_tier_switch_cost.py (17 tests). Every refusal assertion has a sibling asserting the switch still happens where it pays, so a gate that refused everything fails the file.

Tests: 9022 passed, 13 skipped, 0 failed; 3.13 CI reproduce 9016 passed, 19 skipped; ruff clean; all 8 matrix jobs plus lint green on de1d063 before anything shipped.