Skip to content

fix(api): strip MCP image base64 from tool results in the jinja path (#2374) - #2376

Merged
LostRuins merged 1 commit into
LostRuins:concedo_experimentalfrom
Anai-Guo:fix-tool-image-jinja-2374
Aug 2, 2026
Merged

fix(api): strip MCP image base64 from tool results in the jinja path (#2374)#2376
LostRuins merged 1 commit into
LostRuins:concedo_experimentalfrom
Anai-Guo:fix-tool-image-jinja-2374

Conversation

@Anai-Guo

@Anai-Guo Anai-Guo commented Aug 1, 2026

Copy link
Copy Markdown

Problem

Fixes #2374. When a tool / MCP result carries an image, the OpenAI-compatible chat adapter's jinja code path leaves the base64 payload in the rendered prompt as plain text. As reported, a single 1024×1024 jpeg bloats the context by ~120k tokens. The legacy (non-jinja) path already handles this correctly via strip_mcpcontent_of_media, so the behavior is inconsistent between the two paths.

There are two related gaps:

  1. format_jinja never strips the base64 from tool-role string content before rendering, so the whole MCP JSON (including the base64) is templated as text.
  2. sweep_media_from_messages only recognizes image_url items inside a content list, not MCP-style image content blocks ({"type": "image", "data": ...}), so images delivered that way are dropped instead of attached.

Fix

  • format_jinja: strip the base64 from tool-role string content before rendering (reusing the existing strip_mcpcontent_of_media). The image itself is still swept out and attached separately, so nothing is lost — only the multi-KB base64 stops being rendered as text.
  • sweep_media_from_messages: also recognize type == "image" content blocks in a list and attach their data.

Verification

Reproduced both paths with a standalone harness driving the jinja template + sweep logic:

Scenario Before After
Tool content = MCP JSON string with an image prompt rendered 120,209 chars, base64 inline, 1 image attached prompt 227 chars, no base64 in text, 1 image attached
Tool content = list of MCP image blocks 0 images attached (image dropped) 1 image attached

Change is 7 added lines, no deletions, and mirrors the existing legacy-path behavior.

🤖 Generated with Claude Code

…ostRuins#2374)

When a tool/MCP result carries an image, the OpenAI-compatible chat
adapter's jinja code path left the base64 payload in the rendered
prompt as plain text (a single 1024x1024 jpeg bloated the context by
~120k tokens), while the legacy path already stripped it via
strip_mcpcontent_of_media.

- format_jinja now strips the base64 from tool-role string content
  before rendering, matching the legacy path; the image itself is
  still swept out and attached separately.
- sweep_media_from_messages now also recognizes MCP-style image
  content blocks (type == "image") inside a content list, so images
  delivered that way are attached instead of dropped.
@LostRuins
LostRuins changed the base branch from concedo to concedo_experimental August 2, 2026 02:40
@LostRuins
LostRuins merged commit 4423b3a into LostRuins:concedo_experimental Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tool call result of type "image" is processed as text

2 participants