PlanExecuteRunner already meters real LLM token usage during a run and writes it to the OTel span, but that data never reaches the persisted trajectory or the evaluation metrics, so every plan-execute benchmark report shows 0 tokens and no cost.
Where it breaks (current main, e11d1c1b2022db0396364a6d66e24168955a3bb7):
src/agent/plan_execute/runner.py, run() (~lines 171-172): self._meter.input_tokens / self._meter.output_tokens are correctly computed and set on the span (gen_ai.usage.input_tokens / output_tokens).
- A few lines later,
persist_trajectory(...) (~line 184) is called with trajectory=trajectory only, a list[StepResult]. StepResult (src/agent/plan_execute/models.py) has no token field, so the meter's numbers never get passed through.
src/observability/persistence.py, persist_trajectory() (lines 39-79): the persisted record has no top-level token fields either, only run_id/scenario_id/runner/model/question/answer/trajectory.
src/evaluation/metrics.py, _from_plan_execute() (~lines 130-145): builds OpsMetrics from the persisted steps list, and its own comment says why it's incomplete: "plan-execute persists list[StepResult]; the dataclass exposes server/tool/response fields but no per-step token counts, so we surface what is available and leave the rest at zero." _estimate_cost(model, 0, 0) then short-circuits to None (it returns None whenever both token counts are 0).
So every other (SDK-based) runner reports real tokens/cost because their Trajectory dataclass carries per-turn token fields, and plan-execute alone always reports tokens_in=0, tokens_out=0, est_cost_usd=None, even though the correct numbers exist a few lines above the call that drops them.
This looks like the same class of gap already fixed for a different runner: #421 ("OpenCode token usage is reported as zero in evaluation aggregates"), closed by c244f63 in this same metrics.py file.
Fix shape I'd propose (happy to send a PR): thread self._meter.input_tokens / self._meter.output_tokens through persist_trajectory as optional kwargs (backward compatible, the persisted record already tolerates extra fields), and have _from_plan_execute read them instead of hardcoding 0, 0. Flagging as an issue first rather than a surprise PR since it touches the shared persist_trajectory signature and the OpsMetrics construction path used by every runner, and I wasn't sure if you'd rather add the token fields to StepResult/the trajectory dataclass itself instead of passing them alongside it.
Not yet reproduced end-to-end in this session (would need an actual plan-execute run persisted and re-evaluated to show tokens_in/tokens_out/est_cost_usd come back non-zero); the above is from reading the three files against current main and confirming none of the 25 currently open PRs touch persist_trajectory(), _from_plan_execute, or src/evaluation/metrics.py.
PlanExecuteRunneralready meters real LLM token usage during a run and writes it to the OTel span, but that data never reaches the persisted trajectory or the evaluation metrics, so every plan-execute benchmark report shows 0 tokens and no cost.Where it breaks (current
main,e11d1c1b2022db0396364a6d66e24168955a3bb7):src/agent/plan_execute/runner.py,run()(~lines 171-172):self._meter.input_tokens/self._meter.output_tokensare correctly computed and set on the span (gen_ai.usage.input_tokens/output_tokens).persist_trajectory(...)(~line 184) is called withtrajectory=trajectoryonly, alist[StepResult].StepResult(src/agent/plan_execute/models.py) has no token field, so the meter's numbers never get passed through.src/observability/persistence.py,persist_trajectory()(lines 39-79): the persisted record has no top-level token fields either, onlyrun_id/scenario_id/runner/model/question/answer/trajectory.src/evaluation/metrics.py,_from_plan_execute()(~lines 130-145): buildsOpsMetricsfrom the persistedstepslist, and its own comment says why it's incomplete: "plan-execute persistslist[StepResult]; the dataclass exposesserver/tool/responsefields but no per-step token counts, so we surface what is available and leave the rest at zero."_estimate_cost(model, 0, 0)then short-circuits toNone(it returnsNonewhenever both token counts are 0).So every other (SDK-based) runner reports real tokens/cost because their
Trajectorydataclass carries per-turn token fields, and plan-execute alone always reportstokens_in=0,tokens_out=0,est_cost_usd=None, even though the correct numbers exist a few lines above the call that drops them.This looks like the same class of gap already fixed for a different runner: #421 ("OpenCode token usage is reported as zero in evaluation aggregates"), closed by
c244f63in this samemetrics.pyfile.Fix shape I'd propose (happy to send a PR): thread
self._meter.input_tokens/self._meter.output_tokensthroughpersist_trajectoryas optional kwargs (backward compatible, the persisted record already tolerates extra fields), and have_from_plan_executeread them instead of hardcoding0, 0. Flagging as an issue first rather than a surprise PR since it touches the sharedpersist_trajectorysignature and theOpsMetricsconstruction path used by every runner, and I wasn't sure if you'd rather add the token fields toStepResult/the trajectory dataclass itself instead of passing them alongside it.Not yet reproduced end-to-end in this session (would need an actual plan-execute run persisted and re-evaluated to show
tokens_in/tokens_out/est_cost_usdcome back non-zero); the above is from reading the three files against currentmainand confirming none of the 25 currently open PRs touchpersist_trajectory(),_from_plan_execute, orsrc/evaluation/metrics.py.