GPT Vision Feature - Is Base64 PNG Supported?

[NBS-LLC]

I'm trying to use the /v1/chat/completions endpoint with llava via the quickstart command:
docker run -ti -p 8080:8080 localai/localai:v2.8.2-ffmpeg-core llava

If I supply base64 png data the call fails, docker output shows: Failed encoding image not valid string

However if I convert that same png into a jpeg the call will succeed.

Request data looks like (I'm using typescript and axios):

  const payload = {
    model: "llava",
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "What is in the image?",
          },
          {
            type: "image_url",
            image_url: {
              url: `data:image/png;base64,${base64Image}`,
              // url: `data:image/jpeg;base64,${base64Image}`,
            },
          },
        ],
        temperature: 0.9,
      },
    ],
  };

Any ideas?

[xBelladonna]

I've been having similar issues, but I've noticed that it can indeed decode PNG data, even if you lie to the API and put data:image/jpeg;base64,. This is probably an extremely bad idea to do in production. I haven't looked into the code handling this but I'll try to get around to it at some point and see if I can patch this.

[NBS-LLC]

Brilliant! That is a great workaround for the time being. Thank you :)