Aggregated toolcalls at root observation

[dvag-joerg-winter]

Describe your question

This feature is avail in hosted code-evaluators only at generation-level
https://langfuse.com/changelog/2026-07-10-evaluator-tool-calls

Is any kind of aggregation of all toolcalls of all generations in the parent observation available?
Are there already feature requests or plans for that?
It should be useful for evaulating a turn and it's possibly many toolcall generations (and resulting toolcalls)

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

4.1.0

If Langfuse Cloud

No response

SDK and integration versions

OTEL (Collector) ingestion & hosted evaluators

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Aggregating all tool calls across multiple generations within a parent observation is not currently available as a built-in feature for observation-level evaluators. Observation-level evaluators only evaluate individual observations at ingest time and cannot aggregate across multiple observations within a trace.(1)

Current Options

Trace-level evaluators are the recommended approach for aggregation use cases. Trace-level evaluators run on complete traces and can access all observations within that trace, allowing you to aggregate outputs from multiple observations.(2) When you configure a trace-level evaluator, you can map variables to access the full trace context.(1)

For live production traces with aggregation needs:

• Select "Live Traces" as your evaluation target(1)
• Filter traces by name, tags, userId, and other trace-level attributes(1)
• The tool_calls mapping source is available for both observation and experiment targets(1)

Custom scoring via SDK/API is another option for more complex aggregation logic. You can fetch traces programmatically and attach scores after aggregating observation outputs.(2)

Existing Discussions

There is an open GitHub discussion specifically about running evaluations on aggregated observations.(2) This indicates community interest in this capability. You may want to contribute to that discussion or post a feature request at https://langfuse.com/ideas to add your use case for evaluating a turn with multiple tool call generations.

📚 Sources:

1. LLM-as-a-Judge
2. Running an eval on aggregated observations

---

Have another question? Just tag @inkeep.

[CodeCLS]

Hi @dvag-joerg-winter, if you want you can add this as feature request in our Github Discussions, so other people can upvote it.