LLM-as-a-Judge: Add support for filtering traces by input/output content

[Elipeer]

Describe the feature or potential improvement

Currently, when configuring an LLM-as-a-Judge evaluator, the Filter section allows narrowing down evaluation to specific subsets of data by trace name, tags, userId, and other fields. However, there's no documented way to filter traces based on their input or output content directly.

Requested Feature:

Add the ability to filter traces by input/output content in the LLM-as-a-Judge evaluator configuration UI

Thanks!

Additional information

No response

[Elipeer]

Created PR: #11598

[marliessophie]

Thanks for sharing.

We are launching a new eval system that allows targeting observations, see docs. We will not adding new features to trace level evals going forward, only patching the system.

Adding input/output as filter might be technically feasible for observations, but this depends entirely on:

• Memory/processing time during ingestion, where eval decision is made.
• Filter complexity - simple equality checks vs. regex/contains operations on IO

If this is something you'd be interested in, pls let me know and I will adjust this feature request.

[sidhi-ts]

I'm interesting in this for observations.

Currently what we're doing is adding the output fields with "langfuse." prefix so that it gets added to metadata attributes (and to root level metadata) so that we can filter. This results in a lot of duplication. It would be great if we could avoid doing this.