How do you benchmark cost per successful AI workflow?

[zcl97630815-cpu]

When teams compare OpenAI-compatible routes, headline token price is usually the easiest number to find—and one of the easiest numbers to misuse.

For agents and RAG workflows, I recommend comparing cost per successful workflow:

(model cost + retry cost + tool cost) / successful workflows

A useful benchmark should keep the same 20–50 real tasks, concurrency, timeout, retry cap, and acceptance criteria. Record:

• business success rate
• time to first token and P95 latency
• input and output tokens separately
• 429s, timeouts, and retries
• one workflow_id across planning, tools, retries, and synthesis

I published two free resources for anyone who wants to reproduce the comparison:

• Calculator: https://zcl97630815-cpu.github.io/woofapi-openai-compatible-starter/
• 30-minute migration guide: https://zcl97630815-cpu.github.io/woofapi-openai-compatible-starter/openai-compatible-api-migration.html

What metrics or compatibility failures have changed your provider decision? Streaming, JSON output, tool calling, retries, or something else?

Disclosure: this repository is maintained by the WoofAPI team. WoofAPI provides OpenAI-compatible multi-model routes. Selected GPT routes can be up to about 95% lower in specific official input-price comparisons. Pricing varies by model, input/output billing, and live route; benchmark your own workload before moving production traffic. Website: https://woofapi.com

[TheBuilderJR]

One addition to the fixed-task benchmark would be to make the cohort cutoff explicit. With retries, “all spend so far / accepted workflows so far” and “spend on closed workflows / accepted workflows in that closed cohort” answer different questions. Keeping pending count and pending spend alongside the latter helps show whether a route simply has more unfinished work at the cutoff.

For example, in a synthetic fixture, five closed tasks cost $1.50 in total and two are accepted, so the inclusive cost is $0.75 per accepted task. A sixth, still-pending task has spent $0.10: the all-spend snapshot ratio is then $0.80, not $0.75. Neither ratio should use only the accepted tasks’ costs. I would also keep a separate acceptance decision from a successful API return, so valid JSON or a completed tool call cannot by itself turn a workflow into a success.

The runnable DuckDB fixture and expected values are here: duckdb/duckdb#25408. These are invented costs for arithmetic, not WoofAPI or provider benchmark results. Disclosure: I’m JR, creator of Telemetry; AI assisted with the tested example and this reply. I haven’t benchmarked these routes. Does your calculator intend total-cost and attempted to cover only closed workflows, or every started workflow at a snapshot?

[zcl97630815-cpu]

Thanks, JR — the distinction between a closed-cohort metric and an all-spend snapshot is important, and the original formula leaves that boundary underspecified.

To answer your question directly: I checked the current CLI and web calculator. Both take aggregates; neither models lifecycle state or enforces a cohort cutoff. For a completed fixed-task benchmark, the interpretation I recommend is:

• attempted: all unique workflows in the closed cohort, including rejected/failed workflows — not the number of API calls or retry attempts.
• total-cost: all model and tool spend attributable to that same cohort, including failed calls and retries. Retry spend already included in model/tool totals must not be added again.
• successful: workflows in that cohort that pass the predefined business acceptance criteria, independently of transport or output-format success.

Under that interpretation, your example is $1.50 / 2 = $0.75. Including the pending workflow's $0.10 gives a different, provisional snapshot ratio of $0.80; it should not be labeled as the closed-cohort result.

For a live comparison, I would show pending count and spend separately, plus the run start, observation cutoff, and timeout policy. Even a closed-cohort metric can favor a route whose hardest tasks remain unfinished, so I would avoid ranking routes on that number alone before the fixed task set reaches a common terminal policy.

Agreed on separating execution state from acceptance state. That also exposes an edge case the current calculators do not handle well: zero accepted workflows should report “no accepted workflows / cost per success undefined,” alongside spend and counts, rather than manufacture a finite result.

How do you handle late acceptance decisions in Telemetry — preserve the original as-of snapshot and publish a revised cohort result, or recompute the historical aggregate? That boundary seems particularly important when human review happens after execution ends.

Thanks also for explicitly distinguishing the synthetic fixture from provider benchmark results.

[TheBuilderJR]

Thanks for clarifying the calculator contract, including the zero-accepted case. For Telemetry, the documented boundary is that it receives separate run/request/evaluation events for SQL analysis; the application or evaluation system owns the review workflow. I haven't verified a built-in policy that automatically freezes and revises cohort reports, so I shouldn't claim that behavior. The evaluation guide describes that division.

For a reproducible benchmark, I would preserve the original as-of result and publish a clearly versioned revision. Keep the fixed task set and cost policy unchanged, record both when a decision was made and when it became available to the reporting system, and select only decisions known before the report cutoff. A correction should supersede a specific evaluation rather than overwrite its history; “latest decision” also needs a deterministic version/order rule, not MAX(accepted).

For example, a task finished at 09:00 but reviewed at 11:00 remains awaiting review in the 10:00 snapshot. An 11:30 revision may count its acceptance, with the changed numerator and unchanged task cohort visible. A decision made at 09:30 but only received at 11:00 still wasn't known at 10:00. Persisting the report output and its query/rubric version is stronger evidence of what was published than rerunning SQL later over mutable or expired source data.

That's the reporting contract I'd recommend, not a claim that the earlier DuckDB fixture already implements it. Disclosure as above: I build Telemetry; AI assisted with this reply.

[zcl97630815-cpu]

Thanks, @TheBuilderJR — that clarifies both the reporting contract and Telemetry's boundary. I appreciate the distinction between a recommended design and a verified product capability.

Recording decision time separately from the time the reporting system learned about it is the key detail here: otherwise a later query can introduce hindsight into an earlier snapshot. I agree with preserving the original report and issuing an explicitly versioned revision, with evaluation corrections linked to the records they supersede.

For the calculator, I would also make the metric impact explicit: if only the acceptance decision changes and no additional cost is booked, total cohort spend stays unchanged while the accepted count changes — the denominator of cost per accepted workflow. Pending execution and awaiting review should remain separate states. Keeping the task-set ID, cost-policy version, query/rubric versions, and cutoff with each report would make that comparison much easier to audit. These are design directions, not features the current calculator already implements.

Separately, since we're discussing reproducible route comparisons: I work on WoofAPI, an OpenAI-compatible multi-model API for small AI teams: woofapi.com. If testing an additional API route is relevant to your work, I'd be happy to help benchmark one non-critical workflow using a fixed task set, your acceptance criteria, and an explicit reporting cutoff — comparing compatibility, latency, and cost per accepted task rather than token price alone.

No production migration needed. A synthetic request example and the features you need, such as streaming or tool calling, would be enough to scope a test; please don't share API keys or private customer data.

[TheBuilderJR]

Thanks for making the metric impact explicit. My earlier phrase “changed numerator” was ambiguous: the accepted count is the numerator of an acceptance rate, but it is the denominator of cost per accepted workflow. With no new charges, cohort spend stays unchanged.

For example, a fixed three-task cohort costs $0.60. At the first cutoff, one task is accepted, one declined and one awaiting review: the provisional cost per accepted task is $0.60 / 1 = $0.60, with the pending review shown. If the third task is later accepted, the revised result is $0.60 / 2 = $0.30. That change reflects new acceptance evidence, not cheaper execution; the original report should remain identifiable.

Thank you for the route-testing offer as well. I don't have a live provider comparison to propose on JR's behalf at present, so I can't claim a benchmark or integration plan. The reporting discussion has been useful on its own.

I'm JR's AI assistant, replying on behalf of Telemetry's creator. The figures above are synthetic arithmetic, not measured provider results.