Same Evaluator for Live/Dataset Runs but different vars

[aniket-versant]

Describe your question

I created a custom evaluator and want it to run on both live traces and dataset runs. I've created two different evaluator runs for these purposes but my issue is my dataset runs have the ground truth var while my live traces do not. I don't want to map the ground truth var to anything for my live trace evaluator but I don't see an option for this. Is there a way to do what I'm trying to do or do I need to create two different custom evaluators then - one for my dataset and one fro live traces?

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

https://aix-lens-stage.cnbc.com/project/cmkg8fepf003plj07g28lk8zq/evals/new?evaluator=cmkvbhs9400eslj07q37lzmye

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on the documentation, you'll need to create two separate evaluators—one for live traces and one for dataset runs(1).

When you have variables like {{ground_truth}} that only exist in dataset runs but not in live traces, Langfuse doesn't currently support optional variable mapping within a single evaluator configuration(1). The recommended approach is to create two separate custom evaluator prompts:

1. Live trace evaluator: Uses variables like {{input}} and {{output}} without {{ground_truth}}
2. Dataset run evaluator: Uses {{input}}, {{output}}, and {{ground_truth}}

This is the standard pattern when your evaluation criteria differ between live production data (which lacks ground truth) and experimental dataset runs (which include expected outputs)(1).

📚 Sources:

1. how to use Evaluator with problem, answer, and Expected output?

---

Have another question? Just tag @inkeep.