fix: translate OpenAI image_url content into RubyLLM attachments (#51)#53
Closed
obie wants to merge 2 commits into
Closed
fix: translate OpenAI image_url content into RubyLLM attachments (#51)#53obie wants to merge 2 commits into
obie wants to merge 2 commits into
Conversation
A multimodal user message added to the transcript in OpenAI content-array
form (a `text` part plus `{ type: "image_url", image_url: { url: ... } }`)
was passed to RubyLLM's `add_message` verbatim. RubyLLM treats a raw array
of OpenAI content hashes as plain text, so the image was silently dropped —
a vision model received text only and confabulated an answer, with no error.
Add `Raix::MultimodalContentAdapter`, which translates `image_url` parts into
RubyLLM attachments before `add_message`:
- base64 `data:` URIs are decoded into a binary StringIO, because RubyLLM's
`Attachment` does not recognize `data:` URIs and would treat one as a
filesystem path.
- http(s) URLs are passed through as-is (Attachment fetches them).
Anything that is not an array of hashes containing at least one `image_url`
part is returned untouched, so text completions and the Anthropic
cache_control multipart shape are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #51.
Problem
A multimodal user message added to the
transcriptin OpenAI content-array form is silently dropped on the RubyLLM backend:ruby_llm_requestpassed the array straight tochat.add_message(role: :user, content:). RubyLLM treats a raw array of OpenAI content hashes as plain text — it never recognizes theimage_urlpart as an image, so nothing reaches the provider.Fix
New
Raix::MultimodalContentAdaptertranslates OpenAI-style content arrays into aRubyLLM::Contentwith attachments beforeadd_message:data:URIs are base64-decoded into a binaryStringIO. RubyLLM'sAttachmentdoes not recognizedata:URIs — it would classify one as a filesystem path and blow up onFile.binread. Decoding to IO is the same trick RubyLLM uses internally (providers/gemini/media.rb).image_urlpart is returned untouched, so text completions and the Anthropiccache_controlmultipart shape are unaffected.Tests
New
spec/raix/multimodal_content_adapter_spec.rb(6 examples) verifies adata:URI decodes to a realimage/pngattachment (Marcel content-sniffing), http URLs become URL attachments, text survives alongside the image, string-keyed parts work, and non-image content passes through unchanged.Full suite: 113 examples, 0 failures, 1 pending. RuboCop clean.
Note on base branch
This PR is stacked on
fix-function-tool-adapter-rich-schema(PR #52) so both fixes share the same[2.0.5]CHANGELOG section. Merge #52 first; GitHub will retarget this PR tomainautomatically.🤖 Generated with Claude Code