fix(api): make an agent run's cost and tokens reach the trace

[mmabrouk]

The symptom

An agent run shows a correct cost in the playground and nothing in the trace.

Measured on a real stack: ag.metrics.costs.cumulative.* present on 0 of 48,005 spans, root spans and child spans alike. Separately, ag.metrics.tokens.cumulative present on 0 of 8,719 child agent spans.

The two surfaces never read the same field. The playground reads the number the harness streams to the browser. The trace reads a number the platform recomputes from stored span attributes, and that recompute produces nothing for an agent run.

Three causes, fixed here

1. The platform never accepted a reported cost, it only recomputed one

The runner already stamps the harness's own figure as gen_ai.usage.cost, and the SDK stamps it on the workflow root span. No ingest adapter mapped it into the ag.* namespace, so it sat at a raw JSON path that no product surface reads.

It is now mapped to ag.metrics.costs.cumulative.total.

Cumulative, not incremental. record_usage's own docstring says the value is the run's aggregate total, so calling it incremental would misname it and would double count against the runner's own model-call spans if anything ever aggregates over all spans.

The adapter that carries this mapping is named after a vendor but is not vendor gated. AdapterRegistry.extract_features runs every registered adapter over every span unconditionally, and the file already maps our own runner's cache-token attributes. It is in effect the gen_ai.* semantic-convention mapper. The name is misleading and worth changing separately.

2. The roll-up then overwrote it

Measured on the real pipeline: a span reporting $0.42 with one priceable child came out at $0.00021.

The two numbers describe the same spend, not additive spend. The recomputed one is also the lossier estimate, because cost_per_token never receives the cache buckets and so prices cached reads at the full uncached rate. One measured span with 1 uncached prompt token, 25,182 cache-read tokens and 20 completion tokens was priced at $0.000303 against a harness-reported $0.0082.

So the roll-up now fills only spans that report nothing. A cumulative total already present on entry came from the producer, because _set_cumulative is called exactly once per span per pass.

This is not a hypothetical guard. It goes live the moment anything makes ag.meta.response.model present on our spans, which is what makes the recompute succeed at all.

3. The roll-up did not run on the runner's telemetry at all

One agent run's telemetry arrives in two separate OTLP requests. The SDK sends the workflow root span in one. The runner sends its own subtree, headed by an invoke_agent span whose parent id points into the other request.

The tree builder seeded only from spans with parent_id is None. The runner's request contains no such span, so it produced an empty tree and nothing was cumulated. Even the leaf spans had no cumulative value, which is the decisive tell: had the roll-up run at all, a leaf would carry its own incremental value as its cumulative.

A span whose parent is absent from the batch now seeds a tree, because a dangling parent id means "not in this batch" rather than "no parent". Roll-up stays batch local and still does not cross requests; bridging the two requests is a larger design question and is deliberately out of scope.

Verification

• 1,608 passing in OSS unit tests, 1,867 in EE, plus 90 in the tracing tests specifically. 28 skips in both, all "Postgres not reachable".
• Six new tests drive the real ingest sequence rather than mocking it, including the decisive one, test_reported_cost_wins_over_recomputed_child_costs.
• Four new tests cover the widened seeding rule: a parent absent from the batch, a genuinely parentless root behaving exactly as before, disconnected subtrees rolling up independently with no bleed, and a parent cycle terminating rather than looping.

Notes for the reviewer

• Double-counting analysis. Every span has at most one parent and every parent is expanded at most once, so each span appears at most once in the forest and reaches each ancestor along one path. Costs, tokens and errors all use the same shape, so widening the seed adds nodes without adding paths.
• One behavior change worth naming. Children whose start time precedes their parent's now attach. The old single pass sorted by start time and dropped a child whose parent had not yet been seen. Attaching a previously dropped subtree counts it once, not twice.
• The query read path in parsing.py still keys its top-level traces dict off parent_id is None, so the query response shape is unchanged.

Related

Part of a set of three independent fixes for the same reported problem. The other two are the Python SDK streaming fix and the runner usage fix.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview	Aug 3, 2026 8:40pm

[coderabbitai]

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8f59d918-2ac7-4c6d-a298-df67624ff12c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Summary by CodeRabbit

• Bug Fixes
  • Reported GenAI usage costs are now preserved as cumulative totals during ingestion and roll-up.
  • Prevented duplicate cost counting when costs are reported by multiple spans.
  • Improved cost propagation across disconnected or incomplete span trees.
  • Traces with missing parent spans are now handled correctly.
  • Cyclic span relationships no longer cause invalid metric propagation or processing issues.

Walkthrough

The adapter maps reported usage cost to cumulative totals. Span-tree construction now handles batch-local roots and cycles. Cost roll-up preserves producer-reported totals and recomputes costs only when no reported total exists.

Changes

Reported Cost Ingest

Layer / File(s)	Summary
Map reported usage cost api/oss/src/apis/fastapi/otlp/extractors/adapters/logfire_adapter.py, api/oss/tests/pytest/unit/otlp/test_logfire_adapter.py	The adapter maps gen_ai.usage.cost to costs.cumulative.total. Tests verify that no incremental cost metric is emitted.
Build batch-local span forests api/oss/src/core/tracing/utils/trees.py, api/oss/tests/pytest/unit/tracing/utils/test_trees.py	Tree construction treats missing-parent spans as roots, keeps disconnected subtrees independent, and excludes cyclic spans. Tests cover these cases.
Preserve and propagate reported costs api/oss/src/core/tracing/utils/trees.py, api/oss/tests/pytest/unit/otlp/test_reported_cost_ingest.py	Roll-up preserves nonzero producer-reported totals, propagates them to ancestors, avoids double counting, and recomputes totals when no reported cost exists. Ingest tests cover each path.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OTLPSpans
  participant LogfireAdapter
  participant TraceMetricPropagation
  participant SpanMetrics
  OTLPSpans->>LogfireAdapter: parse gen_ai.usage.cost
  LogfireAdapter->>SpanMetrics: store costs.cumulative.total
  SpanMetrics->>TraceMetricPropagation: propagate trace metrics
  TraceMetricPropagation->>SpanMetrics: preserve reported totals or recompute child costs

Loading

Possibly related PRs

• Agenta-AI/agenta#5352: Both changes update span-tree handling and its tests, but this PR addresses reported cost roll-up.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the primary change: making agent run cost and token data reach traces.
Description check	✅ Passed	The description directly explains the telemetry issues, fixes, scope, and test coverage for the changeset.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/agent-run-cost-in-traces

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mmabrouk]

This is one of three independent fixes for the same reported problem: an agent run shows a cost in the playground and nothing in the trace. They can be reviewed and merged separately, in any order.

• fix(sdk): keep the workflow span current across a streamed agent run #5708 — Python SDK: keep the workflow span current across a streamed run, so usage is recorded at all
• fix(api): make an agent run's cost and tokens reach the trace #5709 — API: accept the reported cost at ingest, stop the roll-up overwriting it, and seed the span tree from a batch whose parent lives elsewhere
• fix(runner): stop reporting context-window size as a run's token usage #5710 — runner: stop reporting context-window size as a token total

Live end-to-end verification of all three together on a running stack is in progress, and I will post the result here.

@coderabbitai review

[coderabbitai]

@mmabrouk I will review #5709. I will assess the API changes independently of #5708 and #5710.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[github-actions]

Railway Preview Environment

Preview URL	https://gateway-pr-5709.up.railway.app/w
Project	agenta-oss-clone-spike
Image tag	pr-5709-4325e1c
Status	Deployed
Railway logs	Open logs
Workflow logs	View workflow run
Updated at 2026-08-03T20:50:23.870Z

[mmabrouk]

Live verification found a defect this PR creates. Do not merge yet.

I deployed all three fixes to a running EE stack and drove real agent runs against it. The headline result is good, and one real problem came out of it.

What works

Cost reaches the trace end to end. Before: ag.metrics.costs.cumulative.total on 0 of 49,861 spans. After: present on the workflow root, on invoke_agent, on turn and on chat spans, visible in the trace UI, and returned by the analytics endpoint.

The three fixes agree on cost. The root span and invoke_agent carry the identical number on every run, and no surface adds them together: summing all spans gives 1.99, while analytics returns 0.4978 because it reads roots only, the same under both focus values.

The streaming fix was verified in its exact failure scenario. A run with a 75-second silent tool gap emitted four SSE keepalives, survived them, and still recorded 4,653 tokens and $0.1773 on the workflow span. The Failed to detach context ... created in a different Context errors went from routine, 52 per hour, to zero across all nine runs.

The runner fix showed a clean cutover at restart: 36 of 40 invoke_agent spans in the three hours before were shaped input 0 / output 0 / total ~63,000, and 0 of 14 after.

The defect

invoke_agent now reports exactly twice the real token count. Measured on three runs: 3,843 became 7,686, 3,961 became 7,922, 4,653 became 9,306. On a multi-turn Pi run it was worse, with the span's own value at 7,776 and its cumulative at 15,552. A reader sees a child span carrying twice its parent's tokens in the trace tree.

The runner stamps the run's total usage on invoke_agent as gen_ai.usage.*_tokens, which ingest maps into the incremental bucket, and the same total is already on that span's children. Until this PR the roll-up never ran on the runner's batch at all, so the duplication was invisible. Seeding the tree from a span whose parent is absent from the batch makes the roll-up run, and it adds the parent's own value to its children's.

Cost escapes only by accident, because it maps to the cumulative bucket where the new guard short-circuits.

This is a latent bug in the producer that the seeding fix exposes, not a flaw in the seeding logic. A fix is in progress and will land as a follow-up on this stack. The rule it aims for: exactly one span owns each incremental observation, and a parent's total is either rolled up from its children or carried as an explicit cumulative summary, never repeated as an incremental one.

Two pre-existing problems, not caused by this PR

On the Pi harness the workflow root still shows no cost. It reproduces in batch mode, which the streaming fix never touched, and historically almost no Pi root ever had a cost. So on Pi the trace now has the cost and the playground does not, which is the reported bug inverted.

A run paused for approval records no usage anywhere.

Verification detail

Nothing needed rebuilding; all three fix paths are bind-mounted from the checkout. Python hot-reloads, and the runner runs under tsx with no watcher, so the runner, api, services and both worker containers were restarted. Confirmed by grepping the running containers' filesystems and by the behavioral cutovers above.

No regressions: streaming streams, keepalives fire, tools execute, batch works, no run errored, no new log warnings.

Not verified: nobody clicked run in the playground, so the playground comparison was made against the exact finish frame field it renders rather than the rendered pixel. Only EE, dev, local sandbox.