My evaluator is not running

[Peamolon]

Describe your question

I've set up a LLM as a judge to run some traces that I generated.
The thing is that once I run my evaluator I don't get any results. This is the configuration I am doing

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on your screenshots, I can see a few potential issues with your LLM-as-a-Judge evaluator configuration:

Check Your Variable Mapping

Your evaluation prompt preview shows variables like {{input}}, {{output}}, and {{ground_truth}}, but you need to ensure these are correctly mapped to your trace data(1). In the variable mapping section, you should:

• Map {{input}} to your trace input
• Map {{output}} to your trace output
• Map {{ground_truth}} to your expected output (if applicable)

The system uses JSONPath expressions to extract data from your traces(1). If your data is nested in JSON objects, you may need to use expressions like $.choices[0].message.content to precisely locate it(1).

Verify Your Filters

Your target filter configuration shows filters for "Whole Environment" and a timestamp range(1). Make sure:

• The traces you want to evaluate actually match these filters
• Check the "Preview sample matched traces" section to confirm traces are being selected(1)
• The timestamp range includes your traces (you have December 16th, 2025 set)(1)

Check Evaluator Logs

Navigate to the evaluator logs to see detailed execution status(1). The logs will show:

• Execution status (Completed, Error, Delayed, or Pending)(1)
• Execution trace IDs that you can click to view the complete LLM interaction(1)
• Any retry errors if the evaluator hit rate limits(1)

You can access logs from the evaluator detail page or by clicking "View execution trace" from score tooltips in the trace view(1).

Common Issues

If evaluations are showing as "Delayed", this means they hit rate limits and are being retried with exponential backoff(1). If they show as "Error", click the execution trace ID to see what went wrong(1).

Make sure your LLM connection is properly configured and that the model you selected supports structured output, which is essential for the system to correctly interpret evaluation results(1).

📚 Sources:

1. LLM-as-a-Judge

---

Have another question? Just tag @inkeep.

[Peamolon]

I've checked all suggestions here but I am still not able to run my evaluator

[jannikmaierhoefer]

@Peamolon, have you checked the logs of the evaluator on the evaluator overview list? If you share the link to your Langfuse project I can have a look what the issue might be.

[Peamolon]

Hi @jannikmaierhoefer thanks for replying my thread, at the end I could run my evaluator.. to be honest I didn't do anything different just worked in other things and after 2 hours I recreated my evaluator and the results started to appear