feat(core): preserve multimodal content blocks in provider responses#833
Merged
feat(core): preserve multimodal content blocks in provider responses#833
Conversation
Deploying agentv with
|
| Latest commit: |
989b04b
|
| Status: | ⚡️ Build in progress... |
This was referenced Mar 29, 2026
Update Claude and Pi providers to preserve non-text content blocks (images) in Message.content instead of discarding them via extractTextContent(). This enables multimodal content to flow from provider response through to evaluators. Changes: - Create shared claude-content.ts with toContentArray() and extractTextContent() used by all 3 Claude providers - Update claude-cli, claude-sdk, claude providers to use structuredContent ?? textContent pattern - Add toPiContentArray() to pi-utils.ts for Pi provider - Update pi-coding-agent convertAgentMessage() to preserve structured content - Add 23 unit tests covering content preservation, backward compat, and end-to-end multimodal flow Text-only responses still produce plain strings (no unnecessary wrapping). extractTextContent() remains available for backward compatibility. Closes #818 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Skip ContentImage blocks when Claude's image block has neither valid source.data nor a non-empty url, preventing invalid empty-source blocks in the structured content output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
christso
force-pushed
the
feat/818-provider-preserve
branch
from
4ff1bac to
989b04b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #818
Depends on #828
Summary
Updates Claude and Pi providers to preserve non-text content blocks (images) in
Message.contentinstead of discarding them viaextractTextContent(). This is the critical pipeline change that enables multimodal content to flow from provider response through to evaluators.Changes
New:
claude-content.ts— shared content mappingtoContentArray()andextractTextContent()into a shared module used by all 3 Claude providers (claude-cli, claude-sdk, claude)Updated providers
claude-content.ts(was already usingtoContentArrayfrom #828)toContentArraypattern — image blocks now preserved inContent[]toPiContentArray()topi-utils.ts;convertAgentMessage()now preserves structured contentProviders NOT changed (text-only APIs)
generateText()returns text only)Pattern used
Text-only responses still produce plain strings (no unnecessary wrapping).
Tests
23 new tests in
content-preserve.test.ts:toContentArray: 7 tests (image preservation, format handling, edge cases)extractTextContent: 5 tests (backward compat)toPiContentArray: 4 tests (Pi format handling)extractPiTextContent: 3 tests (backward compat)All 1666 existing tests continue to pass.