useChatRuntime (AI SDK) with Ollama (local LLM): how to handle Attachments

[kzroo]

I'm not sure if any Ollama supported models support file input, but even if they did, I'm sure the current attachment logic with ai-sdk's runtime in assistant-ui won't work. Therefore, if I wanted to resort to extracting the content of attachments, and passing them into the message as context, specifically for ollama, while keeping existing attachment functionality for other providers like openAi and anthropic (which work out of the box), how would I do that?

Essentially i'm trying to support both local LLM usage and the big providers, but i want to keep attachments working for both, using the useChatRuntime (ai-sdk runtime).

anybody able to shed some light here?

[coderabbitai[bot]]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• How to add attachment using external store runtime #1995
• Attachment of pdf or docs #1029
• Please add file upload functionality to message composer attachments #1934
• Using an external database #1918

👤 Suggested Assignees

• zealshah29
• SushekTamrakar
• vincenzodomina
• VandeveldePaul

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[kzroo]

Well after checking out the source and chatting with AI, it seems the answer is this:

• Images: If the model supports vision (e.g., llama3.2-vision), send images as base64 in the message content. The frontend already converts them to data URLs; extract the base64 portion on the backend.

• PDFs and other non-image files: Extract text content and append it to the message text. Ollama doesn't process binary files directly.

• Text files: Decode from the base64 data URL and append the text to the message content.

In summary, in your backend route (e.g. /api/chat/route.ts), when processing message parts:

• Images → include in content array (if vision-capable model)
• PDFs → extract text, append to message text
• Other files → extract text if possible, append to message text