Is it possible to track LLM as judget evaluator cost by experiment ?

[fernando-torres-blip-ai]

Describe your question

I wish to have a dashboard to monitor the cost related to llm as a judge evaluator, splitting by experiment run. Is it possible ?

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[goutamadwant]

You can track the evaluator spend, but a native “group by experiment run” dimension does not appear to be exposed directly today.

LLM-as-a-judge executions are traced separately in the langfuse-llm-as-a-judge environment and include generation cost. A custom dashboard widget can therefore show total judge spend using the Observations view, sum(totalCost), filtered to:

environment = langfuse-llm-as-a-judge

The limitation is attribution: the evaluator execution metadata currently contains the target trace/observation/dataset-item IDs, but not the experiment-run ID/name. For an exact per-run split, the reliable workaround is to:

1. list the run/items with the Experiments API,
2. use each item's traceId,
3. join that to evaluator executions through their target_trace_id metadata,
4. sum the evaluator observations' totalCost.

On self-hosted v4, the relevant interfaces are the Experiments API, Observations API v2, and Metrics API v2:
https://langfuse.com/docs/api-and-data-platform/features/experiments-api
https://langfuse.com/docs/metrics/features/metrics-api

So: overall evaluator-cost dashboard, yes; direct dashboard split by experiment run, not currently without that join. A first-class experiment-run dimension would remove the custom reporting step.