[question] Custom Model Name + No Actual run + Annotation

[kuatroka]

Hi, I want to evaluate different coding agents with different models and create/annotate and evaluate the output manually without any actual model runs or api calls.

I want to give codex/claude code/droid/opencode, etc... any permutation of models for each agent the same prompt and run them > evaluate manually and annotate if they did what I asked + annotate some subjective outcome ( pretty ui or bad ui, etc) > view reporting results after my experiments with different permutations of agent/model + prompt versions.

Basically I want an option to create a custom model name ( not a full working setup, no working api). For example "codex/gpt-5-codex-mini", "codex/gpt-5-codex-high", "cc/opus-4-5" and give them "prompt20-v-1" and "prompt20-v-2" and without running them through promptfoo annotate the result and even skip the output completely or fill it manually.

Is there a way to do it now without creating any new features?

Thanks

[inkeep-ai-assistant[bot]]

Based on the available documentation, you can partially achieve your workflow using existing Langfuse features, though with some limitations:

What you can do now:

1. Prompt Version Control: You can create and manage different prompt versions using labels(1). You can assign custom labels like "prompt20-v-1" and "prompt20-v-2" to different prompt versions(1).

2. Manual Annotation: Langfuse supports human annotation for manual evaluation(2). You can annotate traces, sessions, and observations with scores using the annotation interface(2). To use this feature, you need to create Score Configurations first(2).

3. Custom Score Types: You can create categorical, numerical, or binary score types to capture different aspects of your evaluation, including subjective outcomes like "pretty ui" or "bad ui"(2).

Current limitations:

The main challenge is that Langfuse's current architecture expects actual model runs and API calls. The documentation shows that prompt experiments and evaluations are designed around real LLM providers(3).

There's an open discussion about supporting custom model adapters(4), where a user requested similar functionality for custom applications that don't directly access LLMs(4). The suggested workaround was to "expose your application in an OpenAI schema-compatible format"(4), but this still requires some form of API endpoint.

Potential workaround:

You could manually create traces in Langfuse using the SDKs and populate them with your custom model names and manually generated outputs, then use the annotation features to evaluate them. However, this would require some custom implementation work on your part.

For your specific use case of completely offline evaluation with custom model names and no API calls, you might need to wait for enhanced custom adapter support or implement a workaround using the existing SDK to manually populate traces.

📚 Sources:

1. Prompt Version Control
2. Human Annotation
3. Langfuse Overview
4. feature: support for creation of custom) model adapters

---

Have another question? Just tag @inkeep.

[kuatroka]

Thanks. I was aware of 1 through 3, but the problem is that any run needs an actual call.

I'll check the SDK suggestion.

[jannikmaierhoefer]

@kuatroka, this guide might also be relevant for you as you want to manually annotate traces.