Evaluation Fails When Extracted Generation Parameters Are Not Invoked in Trace

[qa-ailabs]

Describe your question

If the evaluator extracts parameters from the Generation, and this observation is not called within the trace, the evaluation task will fail.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.137.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

I understand your issue. When you configure an LLM-as-a-Judge evaluator to extract parameters from a Generation observation (like {{tools}} mapped to a specific generation named "tools"), the evaluation will fail if that named generation doesn't exist in the trace.(1)

This is expected behavior - the evaluator requires the mapped data to exist in order to populate the evaluation prompt variables.(1) When using JSONPath expressions or object mappings in your evaluator configuration, the system needs to locate the specified observation within the trace structure.(1)

To resolve this:

1. Verify the observation exists: Ensure your application code is actually creating a generation observation with the name you're referencing (in your case, "tools")

2. Check the observation name: The Object Name field in your evaluator mapping must exactly match the name of the generation in your trace

3. Review your trace structure: You can inspect the full trace to confirm which observations are present and verify their names match your evaluator configuration

4. Adjust your mapping: If the generation name varies or might not always be present, you may need to adjust your evaluator's variable mapping strategy to use trace-level data instead of generation-specific data(1)

The error status in your evaluator logs indicates the JsonPath selector couldn't find the referenced generation observation, causing the evaluation to fail.(1) Debug the LLM-as-a-Judge execution traces by filtering for the environment langfuse-llm-as-a-judge in your tracing table to see the specific error details.(1)

📚 Sources:

1. LLM-as-a-Judge

---

Have another question? Just tag @inkeep.

[maxdeichmann]

@qa-ailabs yes, the behavior you describe is correct. What do you expect to happen otherwise?

[Pactortester]

Thank you for your reply.

If no value is found, return null / n / a. ""; this allows the evaluation task to continue instead of throwing an error.

[maxdeichmann]

Ok, in this case i would then "succeed" but not generate a score. We will take this up.