Datasets evaluattion twice

[damazzy]

Describe your question

I have a question for Datasets evaluattion. I set an Evaluator, Its target is New dataset run items. Then I dataset.run_experiment by SDK。 I checked the trace of dataset.Item.Run, There is an amazing happend which is both trace and span got a evaluation score, and they are different, Could you tell me what happend.Wish your response

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on your screenshots, you're experiencing duplicate evaluation scores - one on the trace level (1.00) and another on the span level (0.90) for the same dataset run item.

When you configure an LLM-as-a-Judge evaluator with the target set to "New dataset run items"(1), the evaluator runs on traces created during dataset experiments(1). The evaluator configuration allows you to specify which dataset the evaluator should run on and whether to target new Dataset Runs(1).

When using run_experiment via the SDK(1), the evaluator is triggered to score the traces produced during the dataset run(2). The evaluator then scores those traces(2).

From your screenshots, it appears the evaluator is running twice - once evaluating the entire trace and once evaluating a specific span within that trace. This could happen if:

1. Your evaluator configuration's target filter is matching both the trace and a nested span
2. The evaluator is being triggered at multiple observation levels within the same trace hierarchy

To troubleshoot this, you should check your evaluator's "Target filter" settings in the configuration(1). The sampling configuration targets "all future dataset run items that match these filters"(1).

Additionally, there's a known issue where evaluator traces created via run_experiment may not be properly linked to their corresponding dataset-item run traces in recent SDK versions(3). This could make it harder to debug why duplicate scores are appearing(3).

To resolve this, review your evaluator configuration to ensure it's only targeting the appropriate observation level (trace or span, not both).

📚 Sources:

1. How to use Langfuse-hosted Evaluators on Dataset Runs?
2. Trying to use Dataset for evaluation, but nothing happens when running the evaluation (UI)
3. Evaluator traces not linked to dataset-item runs when using run_experiment

---

Have another question? Just tag @inkeep.

[damazzy]

this is my evaluator configuration


so i could not understand

[Lotte-Verheyden]

Hi @damazzy, could you share a code snippet on how you trigger the experiment via the SDK?

[damazzy]

hi, thanks for your reply,I have read other discussions, It is an issue with ts. It will be right with python。