Feature Request: Custom Logic for Prompt Experiment Scoring

[yixiaoshen-beep]

Describe the feature or potential improvement

Current limitations

• When running datasets with Prompt Experiment, the output is currently scored only by exact match or LLM judge.

Requested feature

• Add support for deterministic custom logic to score outputs in Prompt Experiments.
• Enable users to define rules or conditions for evaluating output.

Examples of possible custom logic:

• Check if all words in a predefined list appear in the output.
• Validate that the output matches a specific format, such as valid JSON.
• Ensure the output contains a required substring or pattern.

This enhancement would enable more flexible and task-specific evaluation of outputs. Thank you!

Additional information

No response

[yixiaoshen-beep]

It would be more useful if it has even more capabilities during judgement, like calling an external API

[marliessophie]

Agree, supporting templated evaluators (similar to the examples you shared) is something we have on the horizon. Will keep thread open and update when we launch this.

[marliessophie]

Hey, we added support for this as of: https://github.com/langfuse/langfuse/releases/tag/v3.176.0. Live on cloud!