fix(agent): send user image attachments as image blocks, not text (#5359)

[YellowSnnowmann]

Summary

• Fixes the agent being unable to read attached images: PNG failed entirely and JPEG was described inaccurately.
• Lifts inline [IMAGE:data:<mime>;base64,…] markers out of the user text into typed ContentBlock::Image blocks so the provider serializes them as real image_url parts.

Problem

By the time a user turn reaches the tinyagents bridge, the multimodal pipeline has already rehydrated and normalized each attachment into an inline [IMAGE:data:<mime>;base64,…] marker — but chat_message_to_message dropped the whole message into a single ContentBlock::Text. That shipped the base64 to the model as literal text, so vision models never received a real image: PNG (lossless, large) is unrecoverable as text, while JPEG occasionally let the model guess — matching the reported symptom exactly.

Solution

Parse the [IMAGE:…] markers out of the user text and emit a typed ContentBlock::Image per attachment, with the marker-free prose as a leading text block (image-only turns emit no empty text block, which some providers reject). The provider layer already serializes ContentBlock::Image as an image_url part and forwards ImageRef.url verbatim — the marker payload is already a canonical data: URI — so no downstream change is needed. Non-vision models are unaffected: the pipeline never rehydrates a marker for them.

Submission Checklist

• Tests added or updated (happy path + at least one failure / edge case) — mixed text+image turn, image-only turn (no empty text block), multi-image turn, and an unchanged plain-text turn.
• Diff coverage ≥ 80% — the added tests exercise the changed lines; CI diff-cover (cargo-llvm-cov) enforces the gate.
• Coverage matrix updated — N/A: bug fix, no added/removed/renamed feature row.
• All affected feature IDs from the matrix are listed under ## Related — N/A: bug fix.
• No new external network dependencies introduced — pure in-process message conversion.
• Manual smoke checklist updated — N/A: fixes existing image-attachment behaviour; no new release-cut surface.
• Linked issue closed via Closes #NNN.

Impact

• Rust core; all platforms. Vision-capable models now receive attached images as structured image_url parts. cargo fmt + cargo clippy -p openhuman -- -D warnings clean; unit tests green.

Related

• Closes: Agent cannot accurately read or describe images — JPEG inaccurate, PNG fails entirely #5359
• Follow-up PR(s)/TODOs: none.

---

AI Authored PR Metadata (required for Codex/Linear PRs)

Linear Issue

• Key: N/A
• URL: N/A

Commit & Branch

• Branch: fix/agent-image-content-blocks-5359

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 7 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 56c5805a-fe55-4085-92bd-9c41575b9fc6

📥 Commits

Reviewing files that changed from the base of the PR and between 2576b86 and 2f6b4dc.

📒 Files selected for processing (1)

• src/openhuman/agent/message_convert.rs

📝 Walkthrough

Walkthrough

User-message conversion now recognizes inline image markers, creates typed image blocks with MIME metadata, preserves text ordering, omits empty image-only text blocks, and keeps plain-text behavior unchanged.

Changes

Multimodal message conversion

Layer / File(s)	Summary
Image marker parsing and validation src/openhuman/agent/message_convert.rs	User content is split into prose and ContentBlock::Image blocks. Image MIME types are extracted from valid data: URIs. Tests cover single-image, image-only, multiple-image, MIME, and plain-text cases.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: senamakel

Poem

A rabbit found pictures tucked in a line,
And shaped them as blocks, neat and fine.
Text stayed in order,
Images crossed the border,
With MIME tags that help them shine.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes route PNG, JPEG, and other data-URI images as typed image blocks and add coverage for mixed, image-only, and multiple-image messages.
Out of Scope Changes check	✅ Passed	The changes stay within image attachment parsing and message conversion required by issue #5359.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: converting user image attachments into image content blocks instead of text.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/openhuman/agent/message_convert.rs`:
- Around line 142-160: Update parse_image_markers and the surrounding message
conversion flow to preserve the original ordering of prose and image markers
instead of aggregating all text before images. Return ordered text/image
segments, construct ContentBlock values sequentially in that order, and preserve
the existing no-attachment behavior. Add a test covering prose between two image
markers.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 54a170db-2b4e-4a7e-a36f-c165809bf82c

📥 Commits

Reviewing files that changed from the base of the PR and between 5efb24a and 2576b86.

📒 Files selected for processing (1)

• src/openhuman/agent/message_convert.rs

[greptile-apps]

Greptile Summary

This PR fixes image attachments being silently dropped for vision models by lifting inline [IMAGE:data:…] markers out of the user text string into typed ContentBlock::Image blocks before the message reaches the tinyagents bridge.

• Introduces user_content_blocks() which parses [IMAGE:…] markers in source order, flushing prose segments and emitting one ContentBlock::Image per provider-ready reference (data: URI or http(s) URL), leaving unrecognized markers verbatim as text.
• Adds four targeted unit tests covering mixed text+image, image-only, multi-image, non-data-URI, and plain-text turns; the fast path (no markers) is byte-identical to the old behavior.

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped to one conversion helper, the fast path is byte-identical to the old behavior, and four targeted tests cover the new branches.

The fix is well-contained: one new private function, one call-site change, and no downstream modifications needed. The unterminated-marker and non-provider-ready-marker paths fall back safely to verbatim text. The only nit is a dead-code fallback that uses the original text (with markers) rather than an empty string, but it cannot be triggered by any input reachable through the current parsing logic.

Files Needing Attention: No files require special attention beyond the single changed file.

Important Files Changed

Filename	Overview
src/openhuman/agent/message_convert.rs	Adds user_content_blocks to parse [IMAGE:…] markers into typed ContentBlock::Image blocks; fix is well-reasoned, well-tested, and includes tracing — one minor nit in the unreachable empty-blocks fallback

_{Reviews (2): Last reviewed commit: "fix(agent): preserve image/text order an..." | Re-trigger Greptile}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2576b863e6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[M3gA-Mind]

Reviewed against upstream/main: CI green on every lane that applies to the changed areas, no unresolved review threads, and the diff does what the title and body claim. Checked correctness, blast radius on shared seams, that the behaviour change is pinned by a test rather than only asserted, and i18n/em-dash compliance on any new user-facing string. Nothing blocking found.