API to Fetch Custom LLM-as-a-Judge Evaluator Prompts from Evaluator Library

[nunomachado]

Describe the feature or potential improvement

Problem Statement

Currently, there's a gap when trying to use custom LLM-as-a-judge evaluators (created in the Langfuse UI) with SDK experiments. While the web UI allows creating custom evaluator prompts that work great for asynchronous evaluation of traces, these cannot be easily reused in SDK experiments which require synchronous evaluators.

Proposed Feature

Provide an API/SDK to fetch LLM-as-a-judge evaluator configurations from the Evaluator Library, including:

• metadata (name, description, model configuration)
• evaluation prompt template
• score reasoning prompt
• score range prompt

Benefits

• Single source of truth: Maintain evaluators only in the Langfuse UI
• Consistency: Same evaluation logic for both asynchronous trace evaluation and synchronous SDK experiments
• Reduced maintenance: No need to manually sync prompts between UI and code
• Better DX: Seamless integration between trace-based and experiment-based evaluation workflows

Additional information

No response

[jannikmaierhoefer]

Thanks for adding this here @nunomachado. We are tracking this here: https://github.com/orgs/langfuse/discussions/8241