Dataset Experiment Run - Difficulty Running Judge Prompt for Multiple Questions in a Single Call

[lleviraz]

Describe your question

Hi, I’m running an experiment from the SDK on a dataset with X items, and for each item, and Evaluator is configured.
The challenge is that this evaluator uses a judge prompt that internally evaluates a list of questions for each item before returning the average score of 0 to 1. For example:

`
You are a judge evaluating accuracy questions. For each question about the model result, determine whether the answer is True or False.

Answering rules:

• True: The question is confirmed as accurate and true by the model result.
• False: The question is false, contradicted, or not supported by the model result. If the question makes a factual claim and the model result lacks the information, answer "False".

Model result:
{{llm_result}}

Questions:
{{questions}}
`

When there is a single question passed, it usually works ok, but if there are multiple questions, it becomes unreliable.

The question is if there is a way to have the evaluator process the the questions one by one, and do some average, and only then report the score for that item (I believe it is possible to achieve if i run an external evaluator on the client side, but wanted to avoid that).

Here is the answer from the great LF AI: https://langfuse.com/docs/ask-ai?conversationId=conv_01kes99zngf7ab9gn8m1p802ea

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.66.1 OSS

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Unfortunately, the Langfuse-hosted LLM-as-a-Judge evaluators don't currently support iterating over a list of questions within a single evaluator to compute an average score(1).

The current architecture processes each dataset item as a single evaluation pass—the evaluator prompt is populated with your mapped variables (like {{llm_result}} and {{questions}}), sent to the LLM once, and expects a single score and reasoning in return(1).

Your options:

1. External evaluator (client-side): As you mentioned, you can implement custom scoring logic via the SDK that iterates through questions, calls the judge for each, averages the results, and then submits the final score back to Langfuse. See the Custom Scores documentation for guidance on this approach(1).

2. Restructure your dataset: Instead of passing multiple questions per item, you could create separate dataset items for each question. This would allow the evaluator to process each question individually, though you'd lose the automatic averaging per original item.

3. Single-pass prompt engineering: Refine your judge prompt to be more reliable when handling multiple questions in one call—though you've noted this is already unreliable, so this may not fully solve your issue.

The feature you're looking for (iterative evaluation with aggregation within the Langfuse-hosted evaluator) isn't available in the current LLM-as-a-Judge implementation(1). If you need this functionality, the external evaluator approach on the client side is your best path forward.

📚 Sources:

1. LLM-as-a-Judge

---

Have another question? Just tag @inkeep.

[nikhildigde]

@marcklingen for viz

[jannikmaierhoefer]

Hi @lleviraz, thanks! Yes, inkeep is correct. This is not yet suppored.

[lleviraz]

Ok, thanks.

…

On Tue, Jan 13, 2026, 15:20 Jannik Maierhöfer ***@***.***> wrote: Hi @lleviraz <https://github.com/lleviraz>, thanks! Yes, inkeep is correct. This is not yet suppored. — Reply to this email directly, view it on GitHub <#11499 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEQFPH4MQIWTYZUGGPSUHW34GTWLFAVCNFSM6AAAAACRNRVEUGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNBYGU3DINQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nikhildigde]

@jannikmaierhoefer any plans to support such a requirement in the near future? We can contribute if the requirement makes sense in general for the product

[jannikmaierhoefer]

Thanks @nikhildigde, we are currently rebuilding the LLM-as-a-Judge logic in Langfuse, so it might make sense to talk about a potential contribution once this is live in the next couple of weeks.

[nikhildigde]

Sure, thank you. Can you please point us to the changes expected here ? Nikhil Digde

…

On Wed, 14 Jan, 2026, 3:43 pm Jannik Maierhöfer, ***@***.***> wrote: Thanks @nikhildigde <https://github.com/nikhildigde>, we are currently rebuilding the LLM-as-a-Judge logic in Langfuse, so it might make sense to talk about a potential contribution once this is live in the next couple of weeks. — Reply to this email directly, view it on GitHub <#11499 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJICMPCN56J4BYSP6YJM4LD4GYJE7AVCNFSM6AAAAACRNRVEUGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNBZGQ2DAMA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jannikmaierhoefer]

@nikhildigde, we dont have any specs written down publicly yet. But the high-level plan is to make the evaluation decision based on the observation level and not the trace level anymore.

[nikhildigde]

Ok thanks Nikhil Digde

…

On Wed, 14 Jan, 2026, 4:14 pm Jannik Maierhöfer, ***@***.***> wrote: @nikhildigde <https://github.com/nikhildigde>, we dont have any specs written down publicly yet. But the high-level plan is to make the evaluation decision based on the observation level and not the trace level anymore. — Reply to this email directly, view it on GitHub <#11499 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJICMPEHUX2Y6RBHSGVUFYT4GYMYFAVCNFSM6AAAAACRNRVEUGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNBZGQ3DKMA> . You are receiving this because you were mentioned.Message ID: ***@***.***>