How to Evaluate an External API

[ayub-kokabi]

I have a translation API that works in the background using a prompt and leveraging OpenRouter to translate user text:

curl -X POST 'https://sample-translator.com/api/translation/' \
  --header 'Authorization: token TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{
  "content":"Hello World",
  "to": "fa"
}' \
  --header 'Authorization: Token TOKEN'

The response is a JSON object with this structure:

{
  "result": "سلام دنیا"
}

I have a dataset named Sample in the Langfuse datasets section, which has two columns:

• Input is like this:

{
    "content":"How are you?",
    "to": "fa"
}

• Expected output is a string:

"حالت چطوره؟"

Now I want to evaluate this service using Langfuse.
That is, every time I change the background prompt of the product, I want it to run an evaluation on the dataset I have already prepared.

The model google/gemini-2.0-flash-lite-001 on OpenRouter will act as a judge, taking the API translation output along with the expected output and the original text, and give a score based on a prompt. This way, I can see whether the performance has improved, worsened, or stayed the same.

---

Prompt:

Evaluate the quality of the translated text compared to the source text and the expected translation.
Give a score between 0 (completely wrong) and 1 (perfect), considering accuracy, fluency, and style.
Provide a short reason in 1–2 sentences.

Input:
Source: {source_text}
Translation: {translated_text}
Expected: {expected_translation}

Output (JSON):

{
  "score": <0-1>,
  "feedback": "<short explanation>"
}

---

The more I read Langfuse documentation, the more confused I get.
Right now, I don’t know how to use the Prompts section, LLM as a judge, Run experiment via user interface, etc., to actually perform this evaluation.

What are the best practices for evaluating this system?

I also have more complex scenarios, but I wanted to start with this one first to get into Langfuse and then move on to those sections.

[inkeep-ai-assistant[bot]]

To evaluate your external translation API with Langfuse and track the impact of prompt changes, follow these best practices using the built-in features:

1. Prepare Your Dataset

You already have a dataset named Sample in Langfuse. Ensure:

• Each dataset item’s input matches the API call payload (e.g., {"content": "...", "to": "fa"}).
• The expected output is the correct translation as a string(1).

2. Create or Update Your Prompt

In Langfuse Prompt Management, make sure your prompt variables match the dataset item keys. For your use case, you’ll want variables like {{content}} and {{to}}. This ensures smooth mapping when running experiments(1).

3. Run Your API and Log Outputs to Langfuse

For each dataset item:

• Send the input to your translation API.
• Capture the output (result).
• Record this output as the translation result in Langfuse, associating it with the correct dataset item.

4. Set Up LLM-as-a-Judge Evaluator

• Go to the Evaluators page in Langfuse and create a new LLM-as-a-Judge evaluator(2).
• Use your evaluation prompt (with variables for source, translation, and expected output). Example variable mapping:
  • source_text → dataset input (content)
  • translated_text → API response (result)
  • expected_translation → dataset expected output(2)
• Choose your judge model (e.g., google/gemini-2.0-flash-lite-001). Make sure this model supports structured output (JSON)(2).
• Set the output mapping so the LLM returns both score (between 0–1) and feedback as JSON(2).

5. Configure and Run Prompt Experiment via UI

• In Langfuse UI, navigate to your dataset(1).
• Click “Start Experiment” and configure:
  • Dataset name
  • The prompt version you want to test
  • The LLM connection for generation (if you want to re-run the prompt, but in your case you may only need to evaluate the outputs)
  • Select the LLM-as-a-Judge evaluator you set up(1)
• Trigger the run. Langfuse will process each dataset item, score the translation using your judge prompt, and aggregate results(1).

6. Analyze Results

• After the experiment, view aggregate scores and feedback in the Dataset Runs table.
• Compare runs side-by-side to see the effect of prompt changes on translation quality(1).

Tips & Notes

• If your translation API is run outside Langfuse, ensure you record the results as traces or dataset run outputs in Langfuse.
• For more complex scenarios (multiple judges, categorical scores, etc.), you can create custom evaluators or prompts(2)(3).
• If you encounter issues with specific models (e.g., Gemini on OpenRouter), check for known limitations(4).

References for Deep Dives

• LLM-as-a-Judge Setup(2)
• Experiments via UI(1)
• Evaluation Overview(3)

📚 Sources:

1. Experiments via UI (Prompt Experiments)
2. LLM-as-a-Judge
3. Evaluation Overview
4. bug: bad request when running prompt experiment with Google API

---

Have another question? Just tag @inkeep.

[ayub-kokabi]

If someone takes the time to post their question here and patiently waits for a human response, there's a good chance—about 99%—that they've already tried asking AI chatbots!

[jannikmaierhoefer]

Hi @ayub-kokabi, we have a guide on setting up automated evals as part of your CI/CD pipeline here that automatically gets triggered once you change your application or prompt. Please not that automated triggers for Dataset Experiments in the UI are currently not supported.