issue: Multimodal API Fails to Recognize Base64-Encoded Image in Open WebUI v0.5.10

[ZimaBlueee]

Check Existing Issues

• I have searched the existing issues and discussions.
• I am using the latest version of Open WebUI.

Installation Method

Docker

Open WebUI Version

v0.5.10

Ollama Version (if applicable)

No response

Operating System

centos 7

Browser (if applicable)

No response

Confirmation

• I have read and followed all instructions in README.md.
• I am using the latest version of both Open WebUI and Ollama.
• I have included the browser console logs.
• I have included the Docker container logs.
• I have listed steps to reproduce the bug in detail.

Expected Behavior

The API should analyze the provided base64-encoded image and return a description of its content (e.g., objects, text, or scene details) when prompted with "What’s in this image?".

Actual Behavior

The API responds with:

"I'd be happy to describe the image, but since it doesn't contain any text, I can't identify or transcribe oral words..."

This implies the model did not detect the uploaded image in the request, even though the image is included as a valid base64 payload.

Steps to Reproduce

1. Send a POST request to /api/chat/completions with the following CURL:

curl --location 'https://xxxx/api/chat/completions' \  
--header 'Content-Type: application/json' \  
--header 'Accept: application/json' \  
--header 'Authorization: Bearer sk-xxx' \  
--data '{  
    "model": "qwen2-vl-72b-32k",  
    "messages": [  
        {  
            "role": "user",  
            "content": [  
                {"type": "text", "text": "What’s in this image?"},  
                {"type": "image", "url": "data:image/jpeg;base64,/9j/4AAQSxxxxxxxxxxxxxx"}  
            ]  
        }  
    ],  
    "temperature": 1,  
    "stream": false  
}'  

2. Observe the response: The model claims no image was uploaded.

Logs & Screenshots

{  
    "id": "chat6089cee6-1f63-11f0-b118-a662d58ecc00",  
    "object": "chat.completion",  
    "created": 1745317215,  
    "model": "qwen2-vl-72b-32k",  
    "choices": [  
        {  
            "index": 0,  
            "message": {  
                "role": "assistant",  
                "content": "I'd be happy to describe the image, but since it doesn't contain any text..."  
            },  
            "finish_reason": "stop"  
        }  
    ]  
}  

Additional Information

The base64 string is shortened for readability (full string validated via online decoders).

No server-side error logs were provided by Open WebUI (ensure debug mode is enabled if available).

[yerffejytnac]

Same issue here. Not sure why this was converted to a discussion when it’s clearly a bug

[jmizell]

Your format for images is wrong, base64 image content is represented like this

    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What is in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/png;base64,iVBOxxxxxxxxxxxxxxx"
            }
          }
        ]
      }
    ],

Took some digging when I had the same error. Open webui is using openai's input message scheme, and the url should be an object. Type is image_url.

[ZimaBlueee]

@jmizell

Thank you for your solution; it successfully resolved the issue.

On a separate note, I found something confusing in the official OpenAI documentation. It seems they use input instead of messages in their API calls, as shown in this example:

curl https://api.openai.com/v1/responses \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer $OPENAI_API_KEY" \
 -d '{
   "model": "gpt-4.1",
   "input": [
     {
       "role": "user",
       "content": [
         {"type": "input_text", "text": "what is in this image?"},
         {
           "type": "input_image",
           "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
         }
       ]
     }
   ]
 }'

This is quite puzzling. Does OpenWebUI partially adopt OpenAI's calling conventions?

[jmizell]

Not entirely sure, but looking through the route https://github.com/open-webui/open-webui/blob/main/backend/open_webui/routers/openai.py#L693 and I think mostly they pass through the input array. The endpoints basically just calls the downstream api, so a lot of it is is pass through, with some manipulation of parameters.

[Bob2345de]

From open-webui ui on iphone (safari) uploading an image (upload file, then choosing photo library) works but am getting error "400: 'url' field must be a base64 encoded image" when using the same image (or any other) from a pc browser. Is the upload function from pc browser describing the photo attachment differently? any ideas to fix? Thanks