Skip to content

Restore public cost reporting with cumulative, comparable accounting #118

Description

@MaxGhenis

Context

The public leaderboard cost column is being hidden because the current values are not consistently comparable across models:

  • costs mix provider-reported, reconstructed, fallback-priced, and carried aggregate values;
  • actual Batch spend and synchronous-equivalent estimates are not always separated;
  • models use different validated serving recipes and output chunk sizes; and
  • external response retries and row-repair merges can replace prior usage rows instead of accumulating every attempted request, so some displayed totals can omit real spend.

The underlying cost fields and benchmark artifacts remain available internally. This issue tracks the work required before restoring a public cost surface.

Required work

  • Add a run-level cumulative request ledger that records every attempt, including failed requests, full-response retries, and row repairs.
  • Record actual billed cost separately from standardized synchronous-equivalent cost.
  • Serialize cost provenance: provider or estimator, pricing snapshot date and source, pricing mode, estimate flag, input/output/reasoning tokens, cache reads/writes, request count, transport (sync or Batch), response contract, and outputs per request.
  • Decide and document the public metric: operational cost under each model's validated recipe, identical-request intrinsic efficiency, or clearly labeled separate metrics.
  • Re-onboard GPT-5.5 after Give unconfigured gpt-5.5 thinking-class completion headroom #101 using the whole-scenario tool contract and 16,384 completion-token cap. Run a two-household smoke test, then rerun all 100 households if it converges. If whole-scenario serving fails, document and validate the smallest convergent chunk size.
  • Reconcile or explicitly label aggregate-only Fable cost provenance, the Grok fallback estimate, and Sonnet 5 standard versus introductory pricing.
  • Add a validator that rejects incomplete or internally inconsistent public cost records.
  • Add UI regression coverage for the chosen labels, provenance, and unavailable-cost states.

Acceptance criteria

  • Every displayed model has an explicit cost basis, serving recipe, pricing snapshot, and provenance.
  • Cumulative run totals reconcile to the complete request ledger, including attempts later replaced by retries or repairs.
  • Actual billed costs and standardized estimates are never conflated.
  • The GPT-5.5 rerun is complete, audited, and incorporated.
  • The public UI defines the metric precisely and only restores cost after all current models pass validation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions