Skip to content

plan-execute runner's metered token usage never reaches persisted trajectories, so evaluation reports 0 tokens / no cost #499

Description

@AmirF194

PlanExecuteRunner already meters real LLM token usage during a run and writes it to the OTel span, but that data never reaches the persisted trajectory or the evaluation metrics, so every plan-execute benchmark report shows 0 tokens and no cost.

Where it breaks (current main, e11d1c1b2022db0396364a6d66e24168955a3bb7):

  • src/agent/plan_execute/runner.py, run() (~lines 171-172): self._meter.input_tokens / self._meter.output_tokens are correctly computed and set on the span (gen_ai.usage.input_tokens / output_tokens).
  • A few lines later, persist_trajectory(...) (~line 184) is called with trajectory=trajectory only, a list[StepResult]. StepResult (src/agent/plan_execute/models.py) has no token field, so the meter's numbers never get passed through.
  • src/observability/persistence.py, persist_trajectory() (lines 39-79): the persisted record has no top-level token fields either, only run_id/scenario_id/runner/model/question/answer/trajectory.
  • src/evaluation/metrics.py, _from_plan_execute() (~lines 130-145): builds OpsMetrics from the persisted steps list, and its own comment says why it's incomplete: "plan-execute persists list[StepResult]; the dataclass exposes server/tool/response fields but no per-step token counts, so we surface what is available and leave the rest at zero." _estimate_cost(model, 0, 0) then short-circuits to None (it returns None whenever both token counts are 0).

So every other (SDK-based) runner reports real tokens/cost because their Trajectory dataclass carries per-turn token fields, and plan-execute alone always reports tokens_in=0, tokens_out=0, est_cost_usd=None, even though the correct numbers exist a few lines above the call that drops them.

This looks like the same class of gap already fixed for a different runner: #421 ("OpenCode token usage is reported as zero in evaluation aggregates"), closed by c244f63 in this same metrics.py file.

Fix shape I'd propose (happy to send a PR): thread self._meter.input_tokens / self._meter.output_tokens through persist_trajectory as optional kwargs (backward compatible, the persisted record already tolerates extra fields), and have _from_plan_execute read them instead of hardcoding 0, 0. Flagging as an issue first rather than a surprise PR since it touches the shared persist_trajectory signature and the OpsMetrics construction path used by every runner, and I wasn't sure if you'd rather add the token fields to StepResult/the trajectory dataclass itself instead of passing them alongside it.

Not yet reproduced end-to-end in this session (would need an actual plan-execute run persisted and re-evaluated to show tokens_in/tokens_out/est_cost_usd come back non-zero); the above is from reading the three files against current main and confirming none of the 25 currently open PRs touch persist_trajectory(), _from_plan_execute, or src/evaluation/metrics.py.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions