Skip to content

fix: translate OpenAI image_url content into RubyLLM attachments (#51)#53

Closed
obie wants to merge 2 commits into
fix-function-tool-adapter-rich-schemafrom
51-image-content-rubyllm
Closed

fix: translate OpenAI image_url content into RubyLLM attachments (#51)#53
obie wants to merge 2 commits into
fix-function-tool-adapter-rich-schemafrom
51-image-content-rubyllm

Conversation

@obie

@obie obie commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #51.

Problem

A multimodal user message added to the transcript in OpenAI content-array form is silently dropped on the RubyLLM backend:

c.transcript << { user: [{ type: "image_url", image_url: { url: "data:image/png;base64,..." } }] }
c.chat_completion   # vision model gets TEXT ONLY → confabulates an answer, no error raised

ruby_llm_request passed the array straight to chat.add_message(role: :user, content:). RubyLLM treats a raw array of OpenAI content hashes as plain text — it never recognizes the image_url part as an image, so nothing reaches the provider.

Fix

New Raix::MultimodalContentAdapter translates OpenAI-style content arrays into a RubyLLM::Content with attachments before add_message:

  • data: URIs are base64-decoded into a binary StringIO. RubyLLM's Attachment does not recognize data: URIs — it would classify one as a filesystem path and blow up on File.binread. Decoding to IO is the same trick RubyLLM uses internally (providers/gemini/media.rb).
  • http(s) URLs pass through unchanged (Attachment fetches them).
  • Anything that isn't an array of hashes with at least one image_url part is returned untouched, so text completions and the Anthropic cache_control multipart shape are unaffected.
chat.add_message(role: :user, content: MultimodalContentAdapter.translate(content))

Tests

New spec/raix/multimodal_content_adapter_spec.rb (6 examples) verifies a data: URI decodes to a real image/png attachment (Marcel content-sniffing), http URLs become URL attachments, text survives alongside the image, string-keyed parts work, and non-image content passes through unchanged.

Full suite: 113 examples, 0 failures, 1 pending. RuboCop clean.

Note on base branch

This PR is stacked on fix-function-tool-adapter-rich-schema (PR #52) so both fixes share the same [2.0.5] CHANGELOG section. Merge #52 first; GitHub will retarget this PR to main automatically.

🤖 Generated with Claude Code

obie and others added 2 commits June 4, 2026 11:04
A multimodal user message added to the transcript in OpenAI content-array
form (a `text` part plus `{ type: "image_url", image_url: { url: ... } }`)
was passed to RubyLLM's `add_message` verbatim. RubyLLM treats a raw array
of OpenAI content hashes as plain text, so the image was silently dropped —
a vision model received text only and confabulated an answer, with no error.

Add `Raix::MultimodalContentAdapter`, which translates `image_url` parts into
RubyLLM attachments before `add_message`:
- base64 `data:` URIs are decoded into a binary StringIO, because RubyLLM's
  `Attachment` does not recognize `data:` URIs and would treat one as a
  filesystem path.
- http(s) URLs are passed through as-is (Attachment fetches them).

Anything that is not an array of hashes containing at least one `image_url`
part is returned untouched, so text completions and the Anthropic
cache_control multipart shape are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@obie obie deleted the branch fix-function-tool-adapter-rich-schema June 4, 2026 09:12
@obie obie closed this Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant