Skip to content

feat(pricing): model-aware cost calculation for dashboard#310

Merged
github-actions[bot] merged 2 commits into
mainfrom
spboyer/issue-308-model-aware-cost
Jun 6, 2026
Merged

feat(pricing): model-aware cost calculation for dashboard#310
github-actions[bot] merged 2 commits into
mainfrom
spboyer/issue-308-model-aware-cost

Conversation

@spboyer

@spboyer spboyer commented Jun 6, 2026

Copy link
Copy Markdown
Member

Closes #308

Summary

Replaces the flat $0.00025/token dashboard cost estimate with a three-tier pricing pipeline:

  1. sdk — use the per-request cost the Copilot SDK already reports (ModelUsage.RequestCost), when available.
  2. table — price input / output / cache_read / cache_write tokens against a per-model rate table (Claude 3.5/4.x, GPT‑5 family, GPT‑4o/4.1, Gemini 2.5).
  3. estimate — fall back to the original flat $0.00025/token rate when the model is unknown.

The chosen source is propagated as costSource through RunSummary / SummaryResponse so the dashboard can show an info tooltip explaining how the number was computed.

What changed

Backend

  • New internal/pricing packageRates struct, LookupRates (longest-prefix match with a word-boundary check so gpt-4 doesn't match gpt-40-future), Compute(*UsageStats) → (dollars, source), CombineSources([]string), and a model rate table with effective date 2025-01-01.
  • internal/webapi/store.go, storage_adapter.go — replaced estimateCost with pricing.Compute(models.AggregateUsageStats(...)). Summary() now collects per-run sources and combines them with CombineSources.
  • internal/webapi/types.go — added CostSource string to RunSummary and SummaryResponse.
  • Tests — comprehensive internal/pricing unit tests (SDK path, table path with cache tokens, multi-model aggregation, unknown-model fallback, prefix-boundary safety, CombineSources). New handlers_test cases assert costSource round-trips through the HTTP layer and aggregates correctly when mixed.

Frontend

  • web/src/components/InfoTooltip.tsx — small lucide-react Info icon with a native title tooltip.
  • web/src/lib/format.tscostSourceTooltip(source?) helper centralizes the four wordings (sdk, table, estimate, mixed).
  • web/src/api/client.ts — added optional costSource?: CostSource to RunSummary and SummaryResponse.
  • RunsTable — info icon in the Cost column header; per-row title reveals the source per run.
  • KPICards — info icon next to "Avg Cost" reflecting the aggregate source.
  • RunDetail — info icon next to the Cost stat card.

Docs

  • site/src/content/docs/guides/dashboard.mdx — new "Cost accuracy" section explaining the three sources, when each applies, the rate-table effective date, and a note that the table is best-effort.

Validation

  • go test ./... — all packages pass (internal/pricing, internal/webapi including new costSource tests).
  • go vet ./... — clean. make lint reports golangci-lint not installed, skipping... on this host; CI will run it.
  • cd web && npx tsc -b && npm run build — clean.
  • cd web && npx playwright test --project=chromium52/52 passed (screenshots regenerated to include the new info icons).

Design notes / follow-ups

  • The rate table is intentionally embedded as a Go map for simplicity. The values are best-effort list pricing snapshots; a sensible follow-up would be externalizing them to YAML/JSON with a published effective-date metadata file (or pulling from a provider catalog). I left a note in the dashboard guide.
  • LiveView, CompareView, and TrendsPage show per-run costs that already benefit from the new pricing pipeline (their values come from the same RunSummary). I scoped the tooltip wiring to the three primary surfaces called out in the issue to keep the diff focused; happy to extend if reviewers prefer.
  • resultSummaryToRunSummary in storage_adapter.go still hardcodes Cost: 0 for the ResultSummary legacy path — that struct doesn't carry token data, so there's nothing to price. Left unchanged; CostSource is omitted (omitempty) for those rows.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Replace the flat $0.00025/token cost estimate with a three-tier
pricing pipeline that prefers SDK-reported cost, falls back to a
per-model rate table, and only uses the flat estimate as a last
resort. Surface the cost source through the JSON API and dashboard
tooltips so users know how each value was computed.

Backend:
- New `internal/pricing` package with `Rates`, `LookupRates`,
  `Compute`, `CombineSources`, and a model rate table covering
  Claude 3.5/4.x, GPT-5 family, GPT-4o/4.1, and Gemini 2.5.
  Long-prefix model-ID matching with a word-boundary check to
  avoid e.g. `gpt-4` matching `gpt-40-future`.
- Replace `estimateCost` in webapi store + storage_adapter with
  `pricing.Compute(models.AggregateUsageStats(...))`.
- Add `costSource` field to `RunSummary` and `SummaryResponse`.
- New tests in pricing package and handlers_test covering SDK
  preference, table path with cache tokens, multi-model aggregation,
  unknown-model fallback, and mixed-source surfacing through HTTP.

Frontend:
- New `InfoTooltip` component using lucide-react `Info`.
- `costSourceTooltip()` helper centralizes wording.
- Tooltips on Runs table Cost header (with per-row source),
  KPI "Avg Cost" card, and Run Detail Cost stat.
- Optional `costSource` on `RunSummary` / `SummaryResponse` types.

Docs:
- New "Cost accuracy" section in dashboard guide explaining the
  three sources and the rate-table effective date.

Closes #308

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 6, 2026 09:20
@github-actions github-actions Bot enabled auto-merge (squash) June 6, 2026 09:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR replaces the dashboard’s flat per-token cost estimate with a model-aware pricing pipeline (SDK-reported cost → per-model rate table → flat fallback), and surfaces the chosen costSource end-to-end so the UI can disclose how the cost was derived.

Changes:

  • Added internal/pricing with rate-table lookup + cost computation + source aggregation, and integrated it into internal/webapi summaries.
  • Plumbed costSource through API types/tests and into the web UI (Runs table, KPI cards, run detail) via tooltip affordances.
  • Documented cost accuracy behavior in the dashboard guide and updated embedded web build output.
Show a summary per file
File Description
web/src/lib/format.ts Adds costSourceTooltip() used by UI tooltips.
web/src/components/RunsTable.tsx Adds cost header info icon + per-row cost source hover tooltip.
web/src/components/RunDetail.tsx Adds cost source info icon in the run detail stat card; extends StatCard.
web/src/components/KPICards.tsx Adds cost source info icon to “Avg Cost”; extends Card.
web/src/components/InfoTooltip.tsx Introduces a reusable inline info icon component.
web/src/api/client.ts Adds CostSource union + optional costSource fields in API types.
web/dist/index.html Updates embedded build asset references.
site/src/content/docs/guides/dashboard.mdx Documents “Cost accuracy” sources and caveats.
internal/webapi/types.go Adds CostSource fields to API responses with docs.
internal/webapi/store.go Replaces estimateCost() with pricing.Compute() using aggregated usage.
internal/webapi/storage_adapter.go Aggregates/returns CostSource for summary responses.
internal/webapi/handlers_test.go Adds/updates tests to assert costSource behavior/round-trip.
internal/pricing/pricing.go New pricing engine: rate table + lookup + compute + CombineSources.
internal/pricing/pricing_test.go Unit tests covering SDK/table/estimate paths + prefix-boundary + combining sources.

Copilot's findings

  • Files reviewed: 13/18 changed files
  • Comments generated: 4

Comment thread web/src/lib/format.ts
Comment thread web/src/components/InfoTooltip.tsx Outdated
Comment thread internal/webapi/types.go
Comment thread internal/webapi/types.go
- gofmt internal/webapi/types.go (resolves CI lint + format check)
- Clarify CostSource doc comments to call out legacy/empty case
- Treat missing costSource as 'data unavailable' rather than estimate
- InfoTooltip: use <button> for keyboard a11y; swap text-muted-foreground
  for text-zinc-500 to match existing repo palette

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot merged commit 8910089 into main Jun 6, 2026
6 checks passed
spboyer pushed a commit that referenced this pull request Jun 6, 2026
Resolve conflicts from #310 (cost-source pricing pipeline):
- Combine Credits (premiumRequests) + CostSource fields on RunSummary/SummaryResponse
- Aggregate both totalPremium and pricing.Compute in Summary()
- StatCard/Card components support both title (Credits tooltip) and labelExtra (Cost InfoTooltip)
- Strengthen Playwright assertions to check actual fixture values
@spboyer spboyer mentioned this pull request Jun 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace flat $0.00025/token cost estimate with model-aware pricing

3 participants