fix(inference): price GB300 cached input from the theoretical hit rate when unmeasured / GB300 缺少实测命中率时按理论命中率为缓存输入定价

[Oseltamivir]

Summary

Follow-up to #975. That PR patched the 14 GB300 dynamo-vllm DeepSeek-V4 AgentX points with GB200 hit rates, but the inflated revenue persisted on other models. The production DB shows the same gap across GB300:

GB300 group	rows without any server_*_cache_hit_rate
dynamo-vllm Kimi K3	7 of 7
dynamo-vllm MiniMax M3	6 of 6
dynamo-trt DSv4 / GLM-5.2 / Qwen3.5	6 of 12 / 7 of 14 / 6 of 12

The cause is the same as #975: the GB300 launcher pins srt-slurm builds that do not hand AIPerf the backend worker /metrics URLs, so only the Dynamo frontend gets scraped and no vllm:prefix_cache_* counters are captured. Cache-aware pricing treats the missing rate as 0 hits and bills ~97% cached input at the uncached price. On Kimi K3 that put GB300 at ~$42/GPU/hr against ~$7 for GB200.

Fix. Add pricingCacheHitRate in src/lib/cache-pricing.ts:

• Prefer the measured server rate (unchanged measuredCacheHitRate semantics, so the fix(db): backfill GB300 dynamo-vllm AgentX cache hit rates from GB200 / 用 GB200 实测值回填 GB300 dynamo-vllm AgentX 缓存命中率 #975 backfill still wins for those 14 points).
• When a GB300 point carries no measurement, fall back to the trace's theoretical_cache_hit_rate, clamped to [0, 1]. The allowlist is a single exported set containing only gb300, and a test pins it.
• Every other hardware keeps returning null, so a missing measurement elsewhere stays visible as a data bug rather than being papered over.

Wired into the three pricing paths: tokenRevenuePerGpuHour (inference chart), useInterpolatedTrendData (historical trends, which now also carries hw and theoretical_cache_hit_rate on its lightweight points), and useThroughputData (calculator). theoretical_cache_hit_rate is added to the calculator response allowlist so the fallback survives the trim; the API catalog guard passes.

Verified against live API rows for Kimi K3 (2026-08-31) at normalized pricing, best point per hardware:

Hardware	Before	After
GB300 dynamo-vllm	~$42/GPU/hr	$6.0/GPU/hr
GB200 dynamo-vllm	$6.9	$6.9
B300 vLLM	$8.2	$8.2

Overlay support: applyTokenRevenuePricing runs on both official rows and ?unofficialrun= overlay points, so the fallback applies to both paths. Tooltips are unchanged and continue to show the measured rate only when one exists, plus the theoretical rate.

Still missing, out of scope here: GB200 dynamo-vllm Kimi K3 (13 of 44 rows), B200/B300 TRT MiniMax M3 (22 of 33 each), and one B200 GLM-5.1 tilert row also lack a server hit rate. Those stay at the uncached price by design of the GB300-only allowlist. The durable fix is upstream: give GB300 AgentX recipes a launcher branch on srt-slurm v1.0.45+ so the sweeps measure their own rate.

Test plan

• New tests in cache-pricing.test.ts (measured wins, GB300 fallback, clamp, other hardware null, allowlist pinned to gb300) and token-revenue.test.ts (GB300 revenue with fallback, other hardware unchanged, measured beats fallback)
• Updated benchmark-api-view.test.ts for the new allowlisted key
• bunx vitest run in packages/app: 292 files, 4789 tests pass
• bun run typecheck, oxlint, oxfmt clean; api-route-catalog.test.ts passes
• Recomputed revenue over live Kimi K3 rows with the patched function (table above)

中文说明

#975 的后续修复。该 PR 用 GB200 的命中率修补了 14 个 GB300 dynamo-vllm DeepSeek-V4 AgentX 数据点，但其他模型上的收入仍然虚高。生产库显示 GB300 普遍存在同样缺口：

GB300 分组	缺少任何 server_*_cache_hit_rate 的行数
dynamo-vllm Kimi K3	7 / 7
dynamo-vllm MiniMax M3	6 / 6
dynamo-trt DSv4 / GLM-5.2 / Qwen3.5	6 / 12、7 / 14、6 / 12

原因与 #975 相同：GB300 启动脚本固定使用的 srt-slurm 版本不会把后端 worker 的 /metrics 地址交给 AIPerf，只抓取了 Dynamo 前端，没有采集 vllm:prefix_cache_* 计数器。缓存感知定价将缺失的命中率视为 0，把约 97% 的缓存输入按未缓存价格计费。Kimi K3 上 GB300 因此约为 $42/GPU/hr，而 GB200 约 $7。

修复方式。 在 src/lib/cache-pricing.ts 新增 pricingCacheHitRate：

• 优先使用实测服务端命中率（measuredCacheHitRate 语义不变，fix(db): backfill GB300 dynamo-vllm AgentX cache hit rates from GB200 / 用 GB200 实测值回填 GB300 dynamo-vllm AgentX 缓存命中率 #975 回填的 14 个点仍以回填值为准）。
• 仅当 GB300 数据点没有实测值时，回退到 trace 的 theoretical_cache_hit_rate，并裁剪到 [0, 1]。白名单是仅含 gb300 的单个导出集合，并有测试固定。
• 其他硬件继续返回 null，缺失测量值时仍以数据问题的形式暴露，而不是被掩盖。

已接入三条定价路径：tokenRevenuePerGpuHour（推理图表）、useInterpolatedTrendData（历史趋势，其轻量数据点现在也携带 hw 与 theoretical_cache_hit_rate）和 useThroughputData（计算器）。theoretical_cache_hit_rate 加入计算器响应字段白名单，避免被裁剪；API catalog 守卫通过。

用线上 Kimi K3（2026-08-31）数据按标准化定价验证，各硬件最佳点：GB300 dynamo-vllm 从约 $42/GPU/hr 降到 $6.0，GB200 保持 $6.9，B300 vLLM 保持 $8.2。

Overlay 支持：applyTokenRevenuePricing 同时作用于官方数据行和 ?unofficialrun= overlay 数据点，回退逻辑对两条路径均生效。Tooltip 不变，仅在存在实测值时显示实测命中率，另显示理论命中率。

仍然缺失、不在本 PR 范围： GB200 dynamo-vllm Kimi K3（44 行中 13 行）、B200/B300 TRT MiniMax M3（各 33 行中 22 行）以及 1 行 B200 GLM-5.1 tilert 也没有服务端命中率。按照仅限 GB300 的白名单设计，这些仍按未缓存价格计费。根本修复应在上游为 GB300 AgentX 配方增加使用 srt-slurm v1.0.45+ 的启动分支，让 sweep 自行测量命中率。

---

Note

Medium Risk
Changes cache-aware revenue math on a hardware-specific fallback; scope is narrow (GB300 allowlist) but affects dollar metrics users compare across SKUs.

Overview
Fixes inflated token revenue on GB300 when AgentX sweeps lack scraped server_*_cache_hit_rate telemetry (missing backend /metrics), which had been pricing ~97% cached input at the uncached rate.

Introduces pricingCacheHitRate: still uses measuredCacheHitRate when present; for GB300 only, falls back to theoretical_cache_hit_rate (clamped). Other hardware keep returning null so missing measurements stay visible.

Wires that helper into token revenue (tokenRevenuePerGpuHour), the calculator (useThroughputData, passing hw), and historical trends (useInterpolatedTrendData, with hw + theoretical rate on lightweight points). theoretical_cache_hit_rate is added to the calculator response metric allowlist so it survives trimming.

^{Reviewed by Cursor Bugbot for commit be7955a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
inferencemax-app	Ready	Preview	Sep 3, 2026 4:58pm UTC