Variable substitution not happening for live observation llm evaluators

[sidhi-ts]

Describe your question

I am using LLM-as-a-judge evaluators for live observation analysis.

I have a few variables that get resolved with some span properties. The input and output get substituted fine, but the variable that requires a particular field within "metadata.attributes" gets resolved to empty/nothing.

The evaluator preview page which shows the variable being substituted fine:

Example trace:

Evaluator trace:

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

v3.224.4

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on your screenshots, the JSONPath you're using to extract from metadata appears to be $.attributes["traceloop.association.properties.original_user_query"]. (1)

For observation-level evaluators, variable mapping pulls data from the matched observation's fields like input, output, and metadata. JSONPath expressions are supported for extracting nested data—for example, $.choices[0].message.content or array slices with negative indices like $[-1:]. (1)

A few things to check:

1. JSONPath syntax: Ensure your JSONPath expression correctly matches the structure of your metadata. Looking at your trace screenshot, the metadata appears to have an attributes object with keys like traceloop.association.properties.original_user_query. Your JSONPath should match this exact structure. Try variations like:

   • $.attributes["traceloop.association.properties.original_user_query"]
   • $.attributes.traceloop\.association\.properties\.original_user_query (escaping dots if needed)

2. Preview validation: The evaluator configuration shows a live preview populated with actual data from the last 24 hours that matched your filters. If the preview shows the variable resolving correctly but the actual evaluation doesn't, there may be inconsistencies in how metadata is structured across different observations. (1)

3. Data availability at evaluation time: Observation-level evaluators only see data on the matched observation itself—they do not load sibling or child observations from the same trace. (1)

There's also an open GitHub issue (#13777) describing similar behavior where certain mapped fields resolve to empty values, though that specific case involves experiments rather than live observations. (2)

If the JSONPath appears correct and the metadata structure is consistent, this may warrant filing a bug report with your specific JSONPath expression and metadata structure.

📚 Sources:

1. LLM-as-a-Judge
2. bug: LLM-as-a-Judge evaluator receives empty input, output, and metadata when running experiments with datasets

---

Have another question? Just tag @inkeep.

[sidhi-ts]

I tried both of these:

$.attributes["traceloop.association.properties.original_user_query"]
$.attributes.traceloop\.association\.properties\.original_user_query

And it didn't work.

I remember this being an issue even for filtering of observations where it was noted that the value for "attributes" is in fact a string a not a dict as it is shown in the UI.

[wochinge]

@sidhi-ts How many characters does your metadata have? We skip parsing large metadata during the evaluation which could be the issue here.

[sidhi-ts]

Metadata can be large. ~150k characters. What is the limit after which you stop parsing metadata attributes?
I am able to access root level keys in the metadata object for all the spans (in evaluators) - it's only the fields within metadata.attributes that i've been having problems with.

[wochinge]

It's 500KB:

langfuse/packages/shared/src/utils/json.ts

Line 67 in 39236b9

const { maxSize = 500_000, maxDepth = 3 } = options;

The other assumption would be that the attributes are encoded to JSON twice.
Can you maybe share a minimal example to reproduce?

[sidhi-ts]

Looks like "attributes" key goes from a key:value form to being fused with all of its child keys. This means that the representation we see when configuring the evaluator is different from the representation during evaluator execution.

Configuring evaluator:

Trace of run evaluator:

[wochinge]

@sidhi-ts Flattening is expected behavior. The issue is that your keys contain non-alphanumeric characters which causes the inconsistencies. Recommendation is to not use alpha-numeric keys.

Metadata handling also changed in v4 of Langfuse (which btw is also much faster) - my recommendation would hence be to upgrade. Is that an option for you?