💡 Sonnet 5 Tokenizer Inflation Forecasting for the Token Cost Observatory

[github-actions[bot]]

Summary

Extend the Token Cost Observatory (token_report.sh) with a tokenizer-inflation estimator that applies content-type-aware multipliers to forecast the fleet's real cost under Sonnet 5's new tokenizer — before the introductory pricing expires August 31, 2026. The new tokenizer inflates token counts by 1.0–1.42x depending on content type (27% for code, up to 42% for English prose), meaning the September 1 standard pricing ($3/$15 MTok) combined with tokenizer inflation yields an effective 20–42% cost increase over today's Sonnet 4.6 rates.

Market Signal

Multiple independent analyses converge on a 20–42% effective cost increase for typical code+prose workloads when both the September 1 rate change and tokenizer inflation are factored in:

• Finout (2026): "Claude Sonnet 5 Pricing: The Hidden Costs Behind the 'Cost-Neutral' Launch" — documents the tokenizer multiplier as a stealth cost driver
• SitePoint: "What the Cost Parity Misses" — the headline $2/$10 intro rate obscures that token counts themselves grow
• Medium/Automation Labs: "Sonnet 5 Counts 30% More Tokens. Its Discount Ends Aug 31." — warns of the compounding effect

Anthropic designed the intro pricing window (through Aug 31) specifically to cushion this transition. The fleet's window to benchmark and optimize is NOW — while costs are lower and before the standard rate kicks in.

Source: Finout — Sonnet 5 Hidden Costs, Claude Platform Docs — What's New in Sonnet 5

User Signal

The Token Cost Observatory (#332) and model-pricing.tsv already track per-model costs with effective dates. The Sonnet 5 intro ($2/$10) and standard ($3/$15) pricing rows are committed. But the pricing rows only capture the per-token rate — they don't model the tokenizer inflation, which means current cost projections understate the real Sonnet 5 cost by up to 42%.

Sonnet 5 is already in the TRIAGE and DEEP model chains as a fallback candidate (#1098/#1115). As it earns primary status via canary rings, the tokenizer impact scales fleet-wide. Related: Token Cost Observatory (#332), Effective Token Anomaly Alerting (#565), Batch API Cost Arbitrage (#639).

Technical Opportunity

The token-metrics.sh library already records per-call JSONL token records with workflow and model labels. Adding a content-type classifier (code vs prose vs mixed, derivable from the prompt file path — e.g., prompts/deep-review.md is code-heavy, prompts/synthesize.md is prose-heavy) and a per-content-type inflation multiplier to the weekly report gives actionable forecasts.

The multiplier can be published as a new tokenizer-inflation.tsv file alongside model-pricing.tsv — same effective-dated TSV idiom the project already uses. The _fmt_usd formatter and cost_usd calculator in the existing toolchain handle the rendering.

Assessment

Dimension	Score	Rationale
Feasibility	high	Extends existing token-metrics.sh + model-pricing.tsv with a new multiplier table; no new APIs needed
Impact	high	Prevents up to 42% cost underestimation in fleet-wide token budget planning; actionable before Aug 31 deadline
Urgency	high	Intro pricing window closes Aug 31, 2026 — 52 days to benchmark and optimize

Adversarial Review

Strongest objection: Content-type heuristics (code: 1.27x, prose: 1.35x) are approximations — actual inflation depends on the specific content. Why not use the Anthropic /messages/count_tokens API for ground-truth measurement instead of a heuristic?

Rebuttal: The /messages/count_tokens API gives exact counts but requires replaying every historical prompt through the API — expensive, slow, and requires storing full prompt content (which the fleet doesn't retain). The heuristic approach gives ~80% accuracy at zero API cost. The weekly report already has both the prompt file path (content type proxy) and the model label (old vs new tokenizer), so a calibration loop — compare heuristic predictions against actual Sonnet 5 token counts from live fallback runs — comes for free and refines the multiplier over time.

Suggested Next Step

Add a tokenizer-inflation.tsv with content-type multipliers (keyed by model glob + content-type tag), extend token_report.sh to emit a "projected Sonnet 5 cost" column using the multiplier, and surface the fleet-wide cost delta in the weekly report's step summary. Lock the heuristic multipliers with a regression test (same pattern as the model-pricing regression guard).