Skip to content

Cost and latency accounting integrity #147

Description

@MaxGhenis

From the August 2026 harness audit (accuracy of published cost/latency columns; fix lands with the v2 pipeline).

Symptoms (verified)

  • Null latency exports as 0.0 (GPT-5.5 batch rows show 0.0s, rendering as impossibly fast) and cost_is_estimated is False where the value is in fact estimated.
  • Spend on failed calls and repair rounds is omitted from published per-model cost: Inkling metered $9.834 actual vs $9.1474 published.
  • $64.08 aggregate discrepancy between the frozen report's cost table and live modelStats, from mixing accounting bases (litellm map consulted before PRICE_OVERRIDES in one path).

Fix

  • Latency columns nullable end-to-end; render "—" not 0.0.
  • One documented cost basis: billed rates via PRICE_OVERRIDES everywhere, litellm map only as fallback with the estimated flag set.
  • Cost ledger includes failed + repair spend, published as total-spend; per-household cost states its numerator definition.
    Tests lock each: null latency round-trip, override-before-map ordering, ledger includes a failed call.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions