relayburn-analyze: foundation (pricing + cost + fidelity)#281
relayburn-analyze: foundation (pricing + cost + fidelity)#281willwashburn merged 2 commits intomainfrom
Conversation
Bootstrap the Rust port of @relayburn/analyze with the three foundational modules every higher-level analyzer consumes: pricing tables, per-record cost derivation, and the fidelity summary used to gate aggregations on input quality. - pricing.rs: ReasoningMode/ModelCost/PricingTable, flatten, load_builtin_pricing, load_pricing(override). Bundles the models.dev snapshot via include_str! so the binary stays self-contained. - cost.rs: cost_for_usage / cost_for_turn / lookup_model_rate / sum_costs with the Codex included-in-output override and per-million f64 math matching the TS path. Honors separate-tariff reasoning models and the synthetic-routing lookup fallback. - fidelity.rs: FidelitySummary, empty/summarize/summarize_from_iter, has_minimum_fidelity. Mirrors the TS class-rank ordering. - provider_reattribution.rs: private helper porting the synthetic reattribution rules (fireworks / explicit / hf:) used by lookup_model_rate. 31 tests pass: cost.test.ts and fidelity.test.ts ports plus pricing and provider-reattribution coverage. Closes #266. https://claude.ai/code/session_01BejN8WdZoaNHTzaopffsQz
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ab3dcbbd9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for provider in root.values() { | ||
| let Some(models) = provider.models.as_ref() else { | ||
| continue; | ||
| }; | ||
| for (id, model) in models { |
There was a problem hiding this comment.
This flattening pass iterates HashMap values and later writes each entry by bare model ID, so collisions are resolved by whichever provider/model pair happens to be visited last. In practice, models.dev.json contains many duplicate IDs with different costs (for example, claude-sonnet-4-6 appears under multiple providers with different cache rates), so the resulting PricingTable can change across runs due to randomized HashMap iteration order, leading to unstable and incorrect cost outputs. Please preserve a deterministic ordering (and ideally TS-compatible precedence) when merging duplicates.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch — the bundled snapshot has ~1938 duplicated model IDs across providers, so randomized HashMap iteration would have made cost lookups drift between runs.
Fixed in 87ba4af: switched ModelsDevRoot and ModelsDevProvider.models to indexmap::IndexMap so deserialization preserves JSON insertion order. Iteration now matches the TS Object.values / Object.entries walk, and the last entry per duplicate ID wins (file-order precedence). Added a regression test that runs flatten 10× against a two-provider fixture sharing one ID and asserts the second-provider entry wins each time.
Generated by Claude Code
models.dev.json contains ~1938 model IDs that appear under multiple providers (e.g. claude-sonnet-4-6 under both anthropic and nano-gpt). The TS flatten iterates Object.values / Object.entries in insertion order and lets the last out[id] = entry win, so duplicate-id resolution is deterministic and matches file order. The HashMap-backed parse randomized iteration order, so duplicate resolution drifted across runs — flagged in the codex review on #281. Switch ModelsDevRoot and ModelsDevProvider.models to IndexMap so deserialization preserves JSON insertion order. Added a regression test that runs the flatten 10x against a two-provider fixture sharing one ID and asserts the second-provider entry wins each time. https://claude.ai/code/session_01BejN8WdZoaNHTzaopffsQz
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
2 participants
Closes #266.
Summary
Bootstraps the
relayburn-analyzecrate with the three foundational modules every higher-level analyzer consumes: pricing tables, per-record cost derivation, and the fidelity summary used to gate aggregations on input quality.pricing.rs—ReasoningMode/ModelCost/PricingTable,flatten,load_builtin_pricing,load_pricing(override). The bundledmodels.dev.jsonsnapshot ships embedded viainclude_str!so the binary stays self-contained and the loader performs no I/O.cost.rs—cost_for_usage/cost_for_turn/lookup_model_rate/sum_costswith the Codexincluded_in_outputoverride and per-millionf64math matching the TS path. Honors separate-tariff reasoning models and the synthetic-routing lookup fallback.fidelity.rs—FidelitySummary,empty_fidelity_summary/summarize_fidelity/summarize_fidelity_from_iter,has_minimum_fidelity. Same class-rank ordering as the TS path.provider_reattribution.rs— private helper porting the synthetic reattribution rules (accounts/fireworks/...,synthetic/...,hf:...) thatlookup_model_rateconsults.f64is the canonical USD type; the per-record arithmetic preserves the TS accumulation order so the documented 1e-9 USD precision contract for the future overhead sub-issue is honored.Conformance
Ports the TS test fixtures with native equivalents:
cost.test.ts→ 12 tests (Claude / Codex / synthetic-reasoner pricing, cache-write rate, override semantics, the Codex 11.3% overstatement regression at <1e-9 USD).fidelity.test.ts→ 7 tests including the OpenCode-fixtureunknown == 0regression that loadstests/fixtures/opencode/multi-turn/...throughparse_opencode_session.cargo test -p relayburn-analyze→ 31 passed.cargo test --workspace→ green.Test plan
cargo build -p relayburn-analyzecargo test -p relayburn-analyzecargo test --workspacemodels.dev.jsonparses and exposesclaude-opus-4-7/claude-sonnet-4-6/claude-haiku-4-5https://claude.ai/code/session_01BejN8WdZoaNHTzaopffsQz
Generated by Claude Code