Langfuse Eval workflow

[minkhantkyaw-brillar]

Describe your question

I have a questions and expected outputs pairs file. I would like to run eval comparing the expected outputs and trace outputs and giving a score. I'm circling and cannot find out how I can do it.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.142.0

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Based on your question-expected output pairs file, you can use Langfuse's Experiments via SDK to evaluate traces against expected outputs.(1)

Recommended Approach: Experiments via SDK

The SDK provides an experiment runner that automatically handles evaluation. Here's how to set it up:(1)

Python SDK Example:

from langfuse import get_client, Evaluation

# Initialize client
langfuse = get_client()

# Define your task function (this runs your application)
def my_task(*, item, **kwargs):
    question = item["input"]
    # Your application logic here that produces output
    # ...
    return output

# Define evaluator to compare expected vs actual output
def accuracy_evaluator(*, input, output, expected_output, metadata, **kwargs):
    if expected_output and expected_output.lower() in output.lower():
        return Evaluation(name="accuracy", value=1.0, comment="Correct answer found")
    
    return Evaluation(name="accuracy", value=0.0, comment="Incorrect answer")

# Load your questions and expected outputs
test_data = [
    {"input": "What is the capital of France?", "expected_output": "Paris"},
    {"input": "What is the capital of Germany?", "expected_output": "Berlin"},
]

# Run experiment with evaluation
result = langfuse.run_experiment(
    name="My Evaluation",
    data=test_data,
    task=my_task,
    evaluators=[accuracy_evaluator]
)

print(result.format())

(1)

Key Points:

1. Data format: Your pairs file should contain input and expected_output fields(1)
2. Evaluators: Define custom evaluator functions that receive output and expected_output to compute scores(1)
3. Automatic tracing: The experiment runner automatically creates traces for each execution(1)
4. Multiple evaluators: You can use multiple evaluators simultaneously to compute different scores(1)

The evaluator function returns an Evaluation object with a score value (0.0 to 1.0 for accuracy) that gets attached to each trace.(1)

📚 Sources:

1. Experiments via SDK

---

Have another question? Just tag @inkeep.

[Lotte-Verheyden]

Hi @minkhantkyaw-brillar, you can follow the instructions from Inkeep bot above, or run an experiment via the Langfuse UI:

1. create your dataset: you can upload your questions and expected output pairs as a CSV file
2. make sure the prompt you want to run is added to Langfuse under "prompt management"
3. have your evaluators set up in the Langfuse UI. In your case, there will be an evaluator that compares the output of the trace with the expected output of your dataset item.
4. run an experiment: there is a button on the page of the dataset. Select your prompt and evaluator.
5. after the experiment is finished, you can see the results in the Langfuse UI

[minkhantkyaw-brillar]

I found that out not long after but what I have in mind is discussed in https://github.com/orgs/langfuse/discussions/9080. The output will be already in the dataset and when the evaluator runs, it will score the output which may be manual input or trace output. Because my app is a RAG app and I'm figuring out a way to evaluate the RAG output. Maybe a custom flow with SDK is possible.

[Lotte-Verheyden]

Thanks for clarifying! We don't support that yet, would also recommend using the SDK for now