[Bug]: OpenAI Vision not working

[LedyBacer]

What happened?

When i upload image and try to ask something - an 400 error appears. In other app with same custom endpoint works fine.

Steps to Reproduce

1. Use docker install
2. Edit .env file, select OPENAI_REVERSE_PROXY and OPENAI_MODELS=gpt-3.5-turbo-1106,gpt-4-1106-preview,gpt-4-vision-preview.
3. docker compose up
4. Try to upload image and ask something.

What browsers are you seeing the problem on?

Chrome

Relevant log output

chat-meilisearch | [2023-12-20T18:14:03Z INFO  actix_web::middleware::logger] 172.24.0.4 "GET /indexes/convos/documents/f583d844-ae92-4f19-a2d1-a728c71ed9fe HTTP/1.1" 404 189 "-" "undici" 0.000559
chat-meilisearch | [2023-12-20T18:14:03Z INFO  actix_web::middleware::logger] 172.24.0.4 "POST /indexes/messages/documents HTTP/1.1" 202 138 "-" "undici" 0.002811
LibreChat      | 2023-12-20 18:14:03 error: [MeiliMongooseModel.findOneAndUpdate] Convo not found in MeiliSearch and will index f583d844-ae92-4f19-a2d1-a728c71ed9fe Document `f583d844-ae92-4f19-a2d1-a728c71ed9fe` not found.
chat-meilisearch | [2023-12-20T18:14:03Z INFO  actix_web::middleware::logger] 172.24.0.4 "POST /indexes/convos/documents HTTP/1.1" 202 135 "-" "undici" 0.003479
chat-meilisearch | [2023-12-20T18:14:03Z INFO  index_scheduler::batch] document addition done: DocumentAdditionResult { indexed_documents: 1, number_of_documents: 205 }
chat-meilisearch | [2023-12-20T18:14:03Z INFO  index_scheduler] A batch of tasks was successfully completed.
chat-meilisearch | [2023-12-20T18:14:03Z INFO  index_scheduler::batch] document addition done: DocumentAdditionResult { indexed_documents: 1, number_of_documents: 17 }
chat-meilisearch | [2023-12-20T18:14:03Z INFO  index_scheduler] A batch of tasks was successfully completed.
LibreChat      | 2023-12-20 18:14:03 warn: [OpenAIClient.chatCompletion][stream] API error
LibreChat      | 2023-12-20 18:14:03 warn: [OpenAIClient.chatCompletion][finalChatCompletion] API error
LibreChat      | 2023-12-20 18:14:03 error: [OpenAIClient.chatCompletion] Unhandled error type Error: 400 image: unknown format
LibreChat      | 2023-12-20 18:14:03 error: [handleAbortError] response error and aborting request Error: 400 image: unknown format
chat-meilisearch | [2023-12-20T18:14:03Z INFO  actix_web::middleware::logger] 172.24.0.4 "POST /indexes/messages/documents HTTP/1.1" 202 138 "-" "undici" 0.002342
chat-meilisearch | [2023-12-20T18:14:03Z INFO  index_scheduler::batch] document addition done: DocumentAdditionResult { indexed_documents: 1, number_of_documents: 206 }
chat-meilisearch | [2023-12-20T18:14:03Z INFO  index_scheduler] A batch of tasks was successfully completed.

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[LedyBacer]

Maybe something wrong with my endpoint. Can you provide curl command for vision test?

[fuegovic]

Maybe something wrong with my endpoint. Can you provide curl command for vision test?

The curl command should look something like this:

• replace sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx with your API key
• and replace the following with your image converted to base64

"iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAMAAABEpIrGAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgwnLpRPAAA..."

• curl request to gpt-4-vision-preview:

curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -d '{
  "model": "gpt-4-vision-preview",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe this image."
        },
        {
          "type": "image_base64",
          "image_base64": {
            "base64": "iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAMAAABEpIrGAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgwnLpRPAAA..."
          }
        }
      ]
    }
  ]
}' "https://api.openai.com/v1/chat"

[LedyBacer]

The curl command should look something like this

So for me works only like this:

curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer sk-***" -d '{
  "model": "gpt-4-vision-preview",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Describe this image."
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANS..."
          }
        }
      ]
    }
  ]
}' "https://api.***.com/v1/chat/completions"

if link like https://api.***.com/v1/chat i get this
{"error":{"message":"Invalid URL (POST /v1/chat)","type":"invalid_request_error","param":"","code":""}}

if img type not image_url and etc i get this
{"error":{"message":"Invalid chat format. Content blocks are expected to be either text or image_url type.","type":"invalid_request_error","param":"","code":null}}%

looks like endpoint matching openai api ref

[fuegovic]

So for me works only like this:
...

Just to clarify, the curl command example I gave was just an example. I'm not 100% sure of the method used by librechat to send the image and I can't reproduce your problem using the official API

[LedyBacer]

Ok, I'll try to figure out

[LedyBacer]

so i check official openai js library, and this code works fine with my endpoint

import { OpenAI } from "openai";

const openai = new OpenAI({
  apiKey: "sk-***", // This is the default and can be omitted
  baseURL: "https://api.***.com/v1/",
});

async function main() {
  const response = await openai.chat.completions.create({
    model: "gpt-4-vision-preview",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "What’s in this image?" },
          {
            type: "image_url",
            image_url: {
              url: "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
            },
          },
        ],
      },
    ],
  });
  console.log(response.choices[0]);
}
main();

[danny-avila]

Image URL are not yet supported. Only uploaded base64

[LedyBacer]

anyway this code also works for me

import { OpenAI } from "openai";

const openai = new OpenAI({
  apiKey: "sk-***", // This is the default and can be omitted
  baseURL: "https://api.***.com/v1/",
});

async function main() {
  const response = await openai.chat.completions.create({
    model: "gpt-4-vision-preview",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "What’s in this image?" },
          {
            type: "image_url",
            image_url: {
              url: "data:image/png;base64,iVBORw0KGgoAAAAN...",
            },
          },
        ],
      },
    ],
  });
  console.log(response.choices[0]);
}
main();

{
  message: {
    role: 'assistant',
    content: 'The image shows a graphic icon that appears to be a complex, intertwined knot or'
  },
  finish_reason: 'length',
  index: 0
}

any suggestions?

[LedyBacer]

or what you mean, i cant DnD image in librechat, only paste base64?

[fuegovic]

You can drag and drop an image in LibreChat, that's what I did earlier

[LedyBacer]

I found the problem - my endpoint dosen't support base64 webp

[danny-avila]

Thanks for letting us know! Are you using a public reverse proxy, if so which one (do we could inform other users if they run into the same)

[LedyBacer]

Are you using a public reverse proxy, if so which one

I use GuoGuo proxy, it's based on one-api. A few days ago one-api officially added support for gpt4 vision, so I don't have long to wait :)