Running an eval on aggregated observations

[gadirajupavantiwari]

Describe your question

I want to run an eval on aggregated output of multiple observation outputs in a trace, is it fusible, if yes how?

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Yes, you can run evaluations on aggregated observation outputs within a trace using custom evaluators in the Experiments SDK.(1)

When using the Experiments SDK, you can create custom evaluators that access multiple observations through the trace context. The evaluator function receives the full trace data, allowing you to aggregate outputs from multiple observations.(1)

Here's how to implement this:

Python SDK:

from langfuse import Evaluation

def aggregated_evaluator(*, item, trace, **kwargs):
    """Evaluate aggregated outputs from multiple observations"""
    # Access all observations in the trace
    observations = trace.observations
    
    # Filter and aggregate outputs as needed
    outputs = [obs.output for obs in observations if obs.output]
    aggregated_output = " ".join(outputs)  # or any aggregation logic
    
    # Evaluate the aggregated output
    score_value = your_evaluation_logic(aggregated_output, item.expected_output)
    
    return Evaluation(
        name="aggregated_score",
        value=score_value,
        comment="Evaluated aggregated observations"
    )

result = langfuse.run_experiment(
    name="Aggregated Evaluation",
    data=test_data,
    task=my_task,
    evaluators=[aggregated_evaluator]
)

(1)

JS/TS SDK:

const aggregatedEvaluator = async ({ item, trace }) => {
    // Access all observations in the trace
    const observations = trace.observations;
    
    // Filter and aggregate outputs
    const outputs = observations
        .filter(obs => obs.output)
        .map(obs => obs.output);
    const aggregatedOutput = outputs.join(" ");  // or any aggregation logic
    
    // Evaluate the aggregated output
    const scoreValue = yourEvaluationLogic(aggregatedOutput, item.expectedOutput);
    
    return {
        name: "aggregated_score",
        value: scoreValue,
        comment: "Evaluated aggregated observations"
    };
};

const result = await langfuse.experiment.run({
    name: "Aggregated Evaluation",
    data: testData,
    task: myTask,
    evaluators: [aggregatedEvaluator]
});

(1)

The key is that evaluators receive the full trace context, which includes all observations, allowing you to implement custom aggregation logic before evaluation.(1)

📚 Sources:

1. Experiments via SDK

---

Have another question? Just tag @inkeep.

[gadirajupavantiwari]

@inkeep , cant we do on live traces

[inkeep-ai-assistant[bot]]

For live production traces, you have two options for evaluating aggregated observation outputs:

1. Trace-level evaluators (supports aggregation across observations)

Trace-level evaluators run on complete traces and can access all observations within that trace, allowing you to aggregate outputs from multiple observations.(1) When you configure a trace-level evaluator, you can map variables to access the full trace context.(1)

Configuration steps:

• Select "Live Traces" as your evaluation target(1)
• Filter traces by name, tags, userId, and other trace-level attributes(1)
• Configure sampling percentage to manage evaluation costs(1)
• Preview matched traces from the last 24 hours to validate your filter configuration(1)

2. Custom scoring via SDK/API

For more complex aggregation logic on live traces, you can use custom scoring through the SDK or API to fetch traces programmatically and attach scores after aggregating observation outputs.(2)

Important limitation: Observation-level evaluators only evaluate individual observations at ingest time and cannot aggregate across multiple observations within a trace.(1) They are designed for operation-level precision, not cross-observation aggregation.(1)

For your use case of aggregating multiple observation outputs, trace-level evaluators are the appropriate choice for live production data.(1)

📚 Sources:

1. LLM-as-a-Judge
2. Troubleshooting and FAQ

---

Have another question? Just tag @inkeep.