Enable images in prompt placeholders

[VildMedPap]

Describe the feature or potential improvement

Currently, the Message Placeholders in Chat Prompts feature only supports pure text content. This prevents developers from fully using the chat prompt templates when working with multimodal models.

Right now, if you want to include images along side the user text input, you can’t rely on Langfuse’s chat templates. Instead, you must fall back to text-only prompt templates and reconstruct the entire chat history and combine with the system prompt. This defeats the purpose of having rich chat templates, reduces maintainability, and makes it more cumbersome to test.

Supporting images in placeholders would allow more flexible and realistic prompt templates for use cases like:

• Visual question answering (e.g. “What’s in this image?”)
• Caption generation or image description tasks
• Multimodal reasoning workflows where user messages may contain images

Current limitation (works fine with text):

from langfuse import Langfuse

# Initialize client
langfuse = Langfuse()

# Get production prompt
prompt = langfuse.get_prompt("no images in placeholder")

msg_history = [
    {"role": "user", "content": "I want to post an image"}
]

# Works fine with text only
compiled = prompt.compile(msg_history=msg_history)

Attempting to include an image fails because placeholders cannot handle the content key being a list:

msg_history = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What’s in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                }
            }
        ]
    }
]

# This results in: AttributeError: 'list' object has no attribute 'find'
compiled = prompt.compile(msg_history=msg_history)

Extend message placeholders to support multimodal content (at least images) alongside text, following the same array-of-blocks structure used in OpenAI/Anthropic APIs.

This would allow developers to use chat templates as intended, rather than being pushed into text-only templates when working with multimodal inputs.

Additional information

No response

[hassiebp]

Hi @VildMedPap - do I understand you correctly that your chat prompt has a placeholder declared at which point you'd like to insert a chat message that includes a content key which is a list of content parts including an image, and this fails?

[TimSusa]

From my side here, I was running to the same issue. Now I am figuring out how to get it done. As far I understood you are supposed to connect your own object storage in order to have the data available in langfuse, is that correct? As I ask, is there a way to put images as base64 to the input dataset? I did not get that done here aswell...

[VildMedPap]

@TimSusa I don't see how your comment is related to my feature request?

[VildMedPap]

Hi @VildMedPap - do I understand you correctly that your chat prompt has a placeholder declared at which point you'd like to insert a chat message that includes a content key which is a list of content parts including an image, and this fails?

@hassiebp just to confirm: this isn’t a bug but rather functionality that hasn’t been implemented yet, correct?

[nimarb]

Hi @VildMedPap , sorry for the late reply. Yes, this is correct - not a bug but missing functionality so far.

[Lucien0907]

Any chance this feature getting implemented?

[joshuaheslin]

+1

Here's a snippet from the TS sdk that would be great if it can be implement. Currently, it's returning a mustache#render error, so I suspect it would be a fairly easy fix.

Desired outcome:
Experiments and datasets can specify content URLs in prompt placeholders so we can evaluate the agent's ability to respond/extract content from the provided file URL.

    const chatPrompt = await langfuse.prompt.get('email-inbox/chat');
    const attachments = [{
      role: 'user',
      content: [{
        type: 'input_file',
        file_url: 'path_to_pdf',
      }]
    }];
    const input = chatPrompt.compile({ emailBody: '...' }, { attachments });

    console.log('input :', input);

throws

TypeError: Invalid template! Template should be a "string" but "array" was given as the first argument for mustache#render(template, view, partials)
    at Object.render (/Users/joshheslin/Development/cloutly/cloutly-backend/node_modules/mustache/mustache.js:753:13)
    at /Users/joshheslin/Development/cloutly/cloutly-backend/node_modules/@langfuse/client/dist/index.cjs:1245:44
    at Array.map (<anonymous>:null:null)
    at _ChatPromptClient.compile (/Users/joshheslin/Development/cloutly/cloutly-backend/node_modules/@langfuse/client/dist/index.cjs:1241:45)
    at Server.<anonymous> (/Users/joshheslin/Development/cloutly/cloutly-backend/src/index.js:158:32)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

[nimarb]

thank you for the report @joshuaheslin , i created an issue with the content and will get back to you once resolved.

[fanta-mnix]

+1 this feature would be deeply appreciated.