Cost is one hardcoded rate per provider, applied to every model and both token directions

[royalpinto007]

Problem

Cost is computed from a single hardcoded number per provider, applied to every model and to every token regardless of direction.

src/providers/openai-compatible.ts:

const costUsd = usage && this.config.costPer1kTokens
  ? (usage.totalTokens / 1000) * this.config.costPer1kTokens
  : undefined;

with the rates baked into the factory functions: openai uses 0.0006, groq uses 0.0001, openrouter uses 0.0006.

Three things are wrong with that:

1. One rate per provider, not per model. The openai factory defaults to gpt-4o-mini, but a suite can specify any model. Run gpt-4o or an o series model through it and the reported cost is off by a large multiple, in the direction that under-reports.
2. Input and output tokens are priced identically. Every major provider charges several times more for output tokens. Using totalTokens with a single rate is structurally wrong, not just imprecise.
3. The numbers are frozen in source. Provider pricing changes. Updating it requires a release of this package.

This feeds the cost gate: the cost scorer decides pass or fail from this number, and open issue #7 notes it passes with a perfect score when no cost is reported. A budget gate is only as good as its inputs, and right now the input is a guess.

Suggested approach

1. Replace costPer1kTokens with a pricing structure keyed by model, with separate input and output rates:

   pricing: { [model: string]: { inputPer1k: number; outputPer1k: number } }

   and compute from promptTokens and completionTokens, which are already parsed out of usage.
2. Let the suite override pricing, so a user on a negotiated rate or a self-hosted endpoint can state their real numbers. That also makes the built-in table a convenience rather than a source of truth.
3. When the model is not in the table, report cost as undefined rather than a wrong number, and make the cost scorer treat unknown cost as an explicit skip or failure rather than a pass. That is the fix cost scorer passes with a perfect score when the provider reports no cost #7 is asking for, and it should be decided together with this.
4. Add a comment next to the built-in table with the date it was checked, and a line in CONTRIBUTING.md about updating it.
5. Consider preferring the provider's own reported cost where one exists. OpenRouter returns usage cost data, so a real number should always beat an estimate.
6. Tests: known model with asymmetric input and output rates, unknown model yielding undefined, suite level override winning over the built-in table.

Done when

• Cost is computed per model with separate input and output rates.
• An unknown model reports unknown cost, never a fabricated one.
• Pricing is overridable from the suite.
• The cost gate's behavior on unknown cost is explicit and documented.

---

If you want to take this on, comment on the issue to claim it and it will be assigned. Please keep to a maximum of 2 open claims per person at a time so other contributors get a chance.