Support for unresolved/literal prompt variables in UI-based Prompt Experiments for custom LLM wrapper APIs

[okaditya84]

Describe the feature or potential improvement

Problem

When running Prompt Experiments via UI with custom LLM wrapper APIs, Langfuse currently resolves all prompt variables (e.g., {{OUT_OF_SCOPE}}) using dataset item values before sending requests to the LLM connection. This prevents wrapper APIs from implementing their own variable substitution logic for constants or predefined values.

Use Case

I have a self-hosted Langfuse instance (v2.60.8) with:

• A chat application that fetches prompts and replaces certain variables (like {{OUT_OF_SCOPE}}) with predefined responses at runtime
• A custom wrapper API configured as an LLM connection for running experiments
• The wrapper API needs to receive the literal {{OUT_OF_SCOPE}} variable so it can apply the same substitution logic used in production

Currently, I must either:

1. Add constant values to every dataset item (redundant and maintenance-heavy)
2. Use SDK-based experiments instead of the UI (loses team collaboration benefits)

Proposed Solution

Add an option to mark certain prompt variables as "literal" or "unresolved" that should be passed through to the LLM connection without substitution. This could be:

• A checkbox/setting per variable in the prompt editor
• A naming convention (e.g., {{CONST_OUT_OF_SCOPE}})
• A configuration in the LLM connection settings

Benefits

• Allows custom wrapper APIs to handle their own variable substitution logic
• Maintains consistency between production prompt handling and experiment execution
• Reduces dataset maintenance burden for constant/predefined values
• Enables UI-based experiments for more complex prompt architectures

Additional information

No response

[jannikmaierhoefer]

Thanks for adding this @okaditya84. As a workaround, have you considered using another naming convention for the variables you want to replace on your LLM wrapper side? E.g., using single brackets: {CONST_OUT_OF_SCOPE}

[okaditya84]

If I try to run an eval from "Run Experiment" option in the datasets, then eval expects to have the dataset item already equipped with the replacement of the variable, if it doesn't then the run experiment will send variable as empty.
I can handle the variable with single braces on the wrapper side, but then it will be a change in format of prompts I have on langfuse prompts section from which the chat app is also fetching the prompts.

Moreover, because single braces wont be recognised as variables in prompts (on langfuse) then if in future I try to update the prompt and mistakenly delete the single braces keyword, then it will cause problems in chat app too.
The thing is double braces are recognized as variables and if I mistakenly remove that while updating the prompt then I will know in the summary section like this:

[jannikmaierhoefer]

Thanks for the context @okaditya84!

[okaditya84]

No problem! But if you can provide a robust turnaround, it will be great and helpful.
@jannikmaierhoefer

Also, just for context: The langfuse version I am using is 2.60.8 and is self-hosted

[okaditya84]

@jannikmaierhoefer Hey I was wondering if there is a way that I receive prompt version that is selected on the "run experiments" side, in my backend wrapper.
I didn't find a documented way to receive the prompt version information in the request body sent to wrapper API during UI-based experiments.
the request body contains prompt text and all, but no version is included.

even if the langfuse is self-hosted.

[jannikmaierhoefer]

@okaditya84, this is currently not implemented. Feel free to add a new ideas post for this.

[okaditya84]

Hey @jannikmaierhoefer
Added this as under Ideas
Kindly refer: https://github.com/orgs/langfuse/discussions/12260
If you allow and assign, I would surely like to work on this issue.