Add functionality to add a regex filter on output of LLM when creating LLM as a judge

[dh-maniceet-sahay]

Describe the feature or potential improvement

Currently we can filter traces on timestamp etc but we can't do any regex filtering on the output of the LLM.
this makes it really difficult to run on past traces where we only wanna judge when a particular output is detected, this results in wastage of tokens and will be a super useful feature (if not already there)

Additional information

No response

[marliessophie]

Hey, could being able to filter based on observation properties rather than trace properties help here? Curious to understand more.

[dh-maniceet-sahay]

So we've a multi-agent system and we finally compose a single response in langgraph which we capture as part of the trace output, hence having this on observation properties will help, but if it can be on trace level that it's even better.

So when we create a LLM as a judge evaluator, then we don't want it to run on all traces, we only want to run on specific traces which have an output pattern. In our case we capture intent through specific structured output, so we can easily check for it using regex or any text filtering mechanism based on the output of trace.

[marliessophie]

I see. This is a gnarly one in our current system. We are working quite extensively on planning next steps for evals, should an avenue open up for this, I will comment here. For the time being however, this is involved to support in the langfuse UI. I'd recommend you play around with custom evaluation pipelines and see if they can help achieve your desired behaviour.