Dataset run items showing identical scores despite unique observation-level scores

[mscherrmann]

Describe your question

I'm experiencing an issue where dataset run items display identical scores across all items, even though the underlying observations have correct, unique score values.

Setup

I'm creating dataset runs by linking observations to dataset items using the API:

langfuse.api.dataset_run_items.create(
    request=CreateDatasetRunItemRequest(
        runName=run_name,
        datasetItemId=item_id,
        observationId=span.id,
        traceId=span.trace_id,
        metadata=run_metadata,
        runDescription=run_description,
    )
)

Before creating the dataset run, I push observation-level scores:

langfuse.create_score(
    trace_id=field_span.trace_id,
    observation_id=field_span.id,
    name=metric_name,
    value=metric_value,
    data_type="NUMERIC",
)

I call langfuse.flush() after pushing all scores and before creating dataset run items.

Dataset run view:

Dataset item run view (individual scores equal to dataset average scores):

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.146.0

If Langfuse Cloud

No response

SDK and integration versions

langfuse-python-sdk version: 3.11.2

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on your description, you're creating observation-level scores correctly and linking them to dataset run items via observationId(1). The issue you're experiencing—where all dataset run items display identical scores despite unique observation-level scores—suggests a potential limitation or bug in how scores are aggregated or displayed at the dataset run item level.

The Langfuse data model supports linking scores to observations within traces(2), and when you create a DatasetRunItem with an observationId, it should associate that specific observation's scores with the run item(2). However, from the screenshots you've provided, it appears the UI is showing aggregated or averaged scores across all items rather than individual observation scores.

Potential causes:

1. Score aggregation behavior: Dataset run views may display aggregated scores (averages) across all items in a run rather than individual observation scores(3). This is particularly true for run-level metrics.

2. Timing/flush issue: Although you're calling langfuse.flush() after creating scores, there may be a delay in score ingestion or propagation to the dataset run item views.

3. Known limitation: There was a related issue where dataset run items were not properly handling multiple run items per dataset item(4), though this was marked as a design limitation rather than a bug. Your case may be similar—the UI might be showing run-level aggregates instead of item-level scores.

Recommendations:

1. Verify observation scores directly: Navigate to the individual traces/observations in the Langfuse UI to confirm the scores are correctly attached at the observation level with unique values.

2. Check score linking: Ensure each score is created with both trace_id and observation_id as you're doing(1):

langfuse.create_score(
    trace_id=field_span.trace_id,
    observation_id=field_span.id,
    name=metric_name,
    value=metric_value,
    data_type="NUMERIC",
)

3. Use Score Analytics: Try using the Score Analytics feature(3) to analyze your scores independently and verify the distribution matches your expectations.

4. API verification: Query scores directly via the API to confirm unique values exist:

# Fetch scores for specific observations
scores = langfuse.api.scores.get(
    trace_id="your_trace_id",
    observation_id="your_observation_id"
)

If observation-level scores are correct but dataset run items still show identical values, this may indicate a display or aggregation issue in the UI for your version (v3.146.0). Consider checking if there are any updates or known issues for your specific version, or reach out to Langfuse support with your specific version details.

📚 Sources:

1. Scores via API/SDK
2. Experiments Data Model
3. Score Analytics
4. bug: Dataset Run Item API running only a single run item for multi-turn dataset item

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @mscherrmann, what is the reason you are creating dataset run items manually and not using the experiment runner sdk or the low-level method?

Manually creating dataset run items is not recommended, as this might cause problems, as a score should be linked both to the observation and the dataset run item.

[mscherrmann]

Thank you for your quick response @jannikmaierhoefer !

The problem is that we currently only have a non-prod deployment of langfuse. On prod, we don't have it deployed yet due to some security concerns. This means we need to be able to turn off langfuse for our evaluations on prod. Relying on the langfuse dataset object is then of course not good.

In addition, using the experiment runner sdk or the low-level method would require a big refactor of our code, since we run our LLM inference on batches of dataset items, not one by one.

Nevertheless this is good to know. Thank you!

[aleksvercau]

I have the same issue, and I am running the experiment with dataset.run_experiment().
Langfuse version: 3.100.0 (self-hosted)
python sdk version: 3.10.6

I ensured that the evaluation metrics are correct prior to upload to langfuse (I checked in the debugger, they are as expected).
Might be related: under scores, all entries have Trace and observation 00000000000:

but metrics are different for each item, so they are correct somewhere inside langfuse.

[aleksvercau]

I found what was the issue in my case: my langfuse client is implemented as a singleton, and it is loaded with tracing disabled. Running an experiment with tracing disabled gives the unique metrics across items apparently. It might be good to raise an error in this scenario (in the sdk or when uploading to langfuse): traces with a score should have a name, id and observation id.

[jannikmaierhoefer]

Thanks for the update on this @aleksvercau!

[so-shiba]

We are running into the same issue.

Our issue is that, we uses Nestjs and BullMQ, and we put the evaluation task into a bullmq that's instrumented with opentelemetry. Using experiment runner sdk will group all the spans under one root trace created by bullmq. Because of this, all our run items show the identical score (which is the average score of all spans of a single root). The cost is wrong too, each experiment item receives the total of cost.

Any solution to this issue?

[jannikmaierhoefer]

Hi @so-shiba, you should be able to filter out the BullMQ spans SDK side as documented here: https://langfuse.com/docs/observability/sdk/advanced-features#filtering-by-instrumentation-scope

This should prevent the observations from being clustered together into one trace. Let me know if this works for you.

[so-shiba]

Thank you for responding. I have tried this but however, all the spans are still grouped together with a root "unnamed" trace.

What criteria does Langfuse use to group the traces?

For more information, we used bullmq otel here

[jannikmaierhoefer]

Hi @so-shiba, thanks. Are you using Langfuse Cloud? If yes, could you please share your project ID - Happy to have a look. In case you are self-hosting Langfuse, please let me know which version you are running.

[so-shiba]

Than you. Our project is hosted at Langfuse Cloud and the project id is cmlkhjhga02y8ad0762fot94d

[Steffen911]

@so-shiba How do you instrument bullmq? Usually, there shouldn't be a trace set by it. Something that could work to sever the parent trace relationship is setting the new span as root like this:

  return getTracer(ctx.traceScope).startActiveSpan(
    ctx.name,
    {
      root: true,
      kind: ctx.spanKind,
    },
    activeContext,
    async (span) => {
    }
)