Research issue: gather examples of multi-modal API calls from different LLMs

[simonw]

To aid in the design for both of these:

• Multi-modal support for vision models such as GPT-4 vision #331
• Design new LLM database schema #556

I'm going to gather a bunch of examples of how different LLMs accept multi-modal inputs. I'm particularly interested in the following:

• What kind of files do they accept?
• Do they accept file uploads, base64 inline files, URL references or a selection?
• How are these interspersed with text prompts? This will help inform the database schema design for Design new LLM database schema #556
• If included with a text prompt does it go before or after the files?
• How many files can be attached at once?
• Is extra information such as the mimetype needed? If so, this helps inform how the CLI design looks (can I do --file filename.ext or do I need some other mechanism that helps provide the type as well?)

[simonw]

Simple GPT-4o example from https://simonwillison.net/2024/Aug/25/covidsewage-alt-text/

import base64, openai

client = openai.OpenAI()
with open("/tmp/covid.png", "rb") as image_file:
    encoded_image = base64.b64encode(image_file.read()).decode("utf-8")
messages = [
    {
        "role": "system",
        "content": "Return the concentration levels in the sewersheds - single paragraph, no markdown",
    },
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": "data:image/png;base64," + encoded_image},
            }
        ],
    },
]
completion = client.chat.completions.create(model="gpt-4o", messages=messages)
print(completion.choices[0].message.content)

[simonw]

Claude image example from https://github.com/simonw/tools/blob/0249ab83775861f549abb1aa80af0ca3614dc5ff/haiku.html

        const requestBody = {
          model: "claude-3-haiku-20240307",
          max_tokens: 1024,
          messages: [
            {
              role: "user",
              content: [
                {
                  type: "image",
                  source: {
                    type: "base64",
                    media_type: "image/jpeg",
                    data: base64Image,
                  },
                },
                { type: "text", text: "Return a haiku inspired by this image" },
              ],
            },
          ],
        };
        fetch("https://api.anthropic.com/v1/messages", {
          method: "POST",
          headers: {
            "x-api-key": apiKey,
            "anthropic-version": "2023-06-01",
            "content-type": "application/json",
            "anthropic-dangerous-direct-browser-access": "true"
          },
          body: JSON.stringify(requestBody),
        })
          .then((response) => response.json())
          .then((data) => {
            console.log(JSON.stringify(data, null, 2));
            const haiku = data.content[0].text;
            responseElement.innerText += haiku + "\n\n";
          })
          .catch((error) => {
            console.error("Error sending image to the Anthropic API:", error);
          })
          .finally(() => {
            // Hide "Generating..." message
            generatingElement.style.display = "none";
          });

[simonw]

Basic Gemini example from https://github.com/simonw/llm-gemini/blob/4195c4396834e5bccc3ce9a62647591e1b228e2e/llm_gemini.py (my images branch):

        messages = []
        if conversation:
            for response in conversation.responses:
                messages.append(
                    {"role": "user", "parts": [{"text": response.prompt.prompt}]}
                )
                messages.append({"role": "model", "parts": [{"text": response.text()}]})
        if prompt.images:
            for image in prompt.images:
                messages.append(
                    {
                        "role": "user",
                        "parts": [
                            {
                                "inlineData": {
                                    "mimeType": "image/jpeg",
                                    "data": base64.b64encode(image.read()).decode(
                                        "utf-8"
                                    ),
                                }
                            }
                        ],
                    }
                )
        messages.append({"role": "user", "parts": [{"text": prompt.prompt}]})

[simonw]

Example from Google AI Studio:

API_KEY="YOUR_API_KEY"

# TODO: Make the following files available on the local file system.
FILES=("image.jpg")
MIME_TYPES=("image/jpeg")
for i in "${!FILES[@]}"; do
  NUM_BYTES=$(wc -c < "${FILES[$i]}")
  curl "https://generativelanguage.googleapis.com/upload/v1beta/files?key=${API_KEY}" \
    -H "X-Goog-Upload-Command: start, upload, finalize" \
    -H "X-Goog-Upload-Header-Content-Length: ${NUM_BYTES}" \
    -H "X-Goog-Upload-Header-Content-Type: ${MIME_TYPES[$i]}" \
    -H "Content-Type: application/json" \
    -d "{'file': {'display_name': '${FILES[$i]}'}}" \
    --data-binary "@${FILES[$i]}"
  # TODO: Read the file.uri from the response, store it as FILE_URI_${i}
done

# Adjust safety settings in generationConfig below.
# See https://ai.google.dev/gemini-api/docs/safety-settings
curl \
  -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro-exp-0801:generateContent?key=${API_KEY} \
  -H 'Content-Type: application/json' \
  -d @<(echo '{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "fileData": {
            "fileUri": "${FILE_URI_0}",
            "mimeType": "image/jpeg"
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Describe image in detail"
        }
      ]
    }
  ],
  "generationConfig": {
    "temperature": 1,
    "topK": 64,
    "topP": 0.95,
    "maxOutputTokens": 8192,
    "responseMimeType": "text/plain"
  }
}')

[simonw]

Here's Gemini Pro accepting multiple images at once: https://ai.google.dev/gemini-api/docs/vision?lang=python#prompt-multiple

import PIL.Image

sample_file = PIL.Image.open('sample.jpg')
sample_file_2 = PIL.Image.open('piranha.jpg')
sample_file_3 = PIL.Image.open('firefighter.jpg')

model = genai.GenerativeModel(model_name="gemini-1.5-pro")

prompt = (
  "Write an advertising jingle showing how the product in the first image "
  "could solve the problems shown in the second two images."
)

response = model.generate_content([prompt, sample_file, sample_file_2, sample_file_3])

print(response.text)

It says:

When the combination of files and system instructions that you intend to send is larger than 20MB in size, use the File API to upload those files, as previously shown. Smaller files can instead be called locally from the Gemini API:

[simonw]

I just saw Gemini has been trained to returning bounding boxes. https://ai.google.dev/gemini-api/docs/vision?lang=python#bbox

I tried this:

>>> import google.generativeai as genai
>>> genai.configure(api_key="...")
>>> model = genai.GenerativeModel(model_name="gemini-1.5-pro-latest")
>>> import PIL.Image
>>> pelicans = PIL.Image.open('/tmp/pelicans.jpeg')
>>> prompt = 'Return bounding boxes for every pelican in this photo - for each one return [ymin, xmin, ymax, xmax]'
>>> response = model.generate_content([pelicans, prompt])
>>> print(response.text)
I found the following bounding boxes:
- [488, 945, 519, 999]
- [460, 259, 487, 307]
- [472, 574, 498, 612]
- [459, 431, 483, 476]
- [530, 519, 555, 560]
- [445, 733, 470, 769]
- [493, 805, 516, 850]
- [418, 545, 441, 581]
- [400, 428, 425, 466]
- [593, 519, 616, 543]
- [428, 93, 451, 135]
- [431, 224, 456, 266]
- [586, 941, 609, 964]
- [602, 711, 623, 735]
- [397, 500, 419, 535]
I could not find any other pelicans in this image.

Against this photo:

It got 15 - I count 20.

[simonw]

I don't think those bounding boxes are in the right places. I built a Claude Artifact to render them, and I may not have built it right, but I got this:

Code here: https://static.simonwillison.net/static/2024/gemini-bounding-box-tool.html

Transcript: https://gist.github.com/simonw/40ff639e96d55a1df7ebfa7db1974b92

[simonw]

Tried it again with this photo of goats and got slightly more convincing result:

>>> goats = PIL.Image.open("/tmp/goats.jpeg")
>>> prompt = 'Return bounding boxes around every goat, for each one return [ymin, xmin, ymax, xmax]'
>>> response = model.generate_content([goats, prompt])
print(response.text)
>>> print(response.text)
- 200 90 745 527 goat
- 300 610 904 937 goat

[simonw]

Oh! I tried different varieties of coordinate and it turned out this one rendered correctly:

[255, 473, 800, 910]
[96, 63, 700, 390]

Rendered:

[simonw]

I mucked around a bunch and came up with this, which seems to work: https://static.simonwillison.net/static/2024/gemini-bounding-box-tool-fixed.html

It does a better job with the pelicans, though clearly those boxes aren't right. The goats are spot on though!