The LLM-as-a-judge evals for dataset items are super sluggish

[pavzagor]

It seems like they are launcher sequentially and that they wait till the previous one finishes.
Would be awesome to trigger them in parallel or speed them up in a different way.

Iterating on the prompts is very difficult because of it.

[inkeep-ai-assistant[bot]]

The sequential evaluation behavior you're experiencing is a known performance consideration with LLM-as-a-Judge evaluators in Langfuse(1).

Currently, the system includes built-in rate limiting and exponential backoff mechanisms to handle LLM provider tier limits(1). When evaluations hit rate limits, they show a "Delayed" status and are retried with exponential backoff(1). This can make the process appear sequential and slow.

For managing evaluation throughput, you can configure sampling to run evaluations on a percentage of matched traces (e.g., 5%) to reduce the total evaluation load(1). Additionally, you can monitor token usage and costs for each evaluator execution through the execution traces(1).

The execution status indicators show(1):

• Completed: Evaluation finished successfully
• Error: Evaluation failed
• Delayed: Evaluation hit rate limits and is being retried with exponential backoff
• Pending: Evaluation is queued and waiting to run

To optimize your prompt iteration workflow, consider using the LLM Playground for initial prompt testing before running full dataset evaluations(1). You can also view detailed logs for each evaluation execution, including the exact prompt sent to the judge LLM and token usage(1).

If you need more control over the evaluation process, you might consider using the SDK approach for dataset experiments, which allows for more customized execution patterns(2).

📚 Sources:

1. LLM-as-a-Judge
2. Experiments via SDK

---

Have another question? Just tag @inkeep.

[jannikmaierhoefer]

Hi @pavzagor, thanks for the feedback, and I agree that parallel executions would offer a better usability here. I'll turn this thread into an "ideas" post to have this on our roadmap.