Running LLM as a judge on single experiments

[matthew-giglio]

Describe your question

Hi all, is it possible to run LLM as a judge evaluators on the result of a given experiment after you've run a given experiment on a dataset? I see that you can wire up evaluators to experiments and have an experiment run the evaluators itself, but I have deterministic scoring in my experiment that does a pretty good first pass at telling me whether the agent performed well, so I only want to think about running LLM as a judge provided the deterministic scoring goes well. However, when I go to the LLM as a judge page, it only lets me choose the whole entire dataset and all its experiment runs rather than just a given experiment to which I want to compare it. It's wasteful to run the LLM as a judge evaluator on a bunch of duplicate experiment runs, so I would love to know if there is a different workflow or upcoming feature that would simplify this UX. Thanks in advance!

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.223.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

The current LLM-as-a-Judge UI for experiments targets datasets and their run items rather than individual experiment runs. (1) This means you cannot directly filter to evaluate only a specific experiment run's results after the fact through the standard evaluator setup flow.

Workaround options:

1. Use observation-level evaluators with metadata filtering: If your experiment traces include distinguishing metadata (like experiment run ID or name), you could set up an observation-level evaluator with filters that target only observations from that specific run. Observation-level evaluators support filtering by trace metadata, tags, and other attributes. (1)

2. Programmatic setup via API: You can set up and manage LLM-as-a-Judge evaluation programmatically through the public API. Evaluation rules define what gets evaluated, including filters, sampling rate, and variable mappings. This could give you more granular control over which data gets evaluated. (1)

3. Ingest scores via SDK: If you want to run LLM-as-a-Judge logic conditionally based on your deterministic scores, you could implement this in your own pipeline and ingest the resulting scores using the Scores via API/SDK approach. (2)

The workflow you're describing—running LLM-as-a-Judge only on experiment runs that pass deterministic checks—isn't directly supported in the current UI. You may want to open a feature request in GitHub Discussions to request the ability to filter evaluators to specific experiment runs.

📚 Sources:

1. LLM-as-a-Judge
2. Code evaluators

---

Have another question? Just tag @inkeep.

[CodeCLS]

Hi @matthew-giglio, this is not currently possible from the UI but should be possible programmatically via our API Endpoints. I have noted this as a request.