[v6.0] fix: LangChain user image conversion returning provider-specific image_url blocks - #17692
Conversation
Bugfix reviewOutcome: approved Fixes issueStatus: fully-addresses URL, data-URL, base64, binary, and image-file inputs now produce canonical LangChain image blocks, and multimodal HumanMessages receive the v1 marker required for downstream provider translation. Side effectsRisk: low The intentional raw message representation change may affect consumers relying on the erroneous image_url shape, while text-only and non-image file behavior remains preserved. PerformanceRisk: none Conversion remains single-pass and retains the existing base64 encoding approach without adding material allocations or repeated processing. Backwards compatibilityRisk: low Existing stored data is not migrated or mutated; only newly converted or persisted multimodal HumanMessages use canonical blocks and output_version v1 metadata. ArchitectureRisk: none The private conversion helper remains owned by the LangChain adapter, consumes LangChain through its declared public messages export, and introduces no cross-package source imports or dependency inversion. Change scopeStatus: minimal All four changed files are directly necessary: one localized implementation change, regression expectation updates at utility and adapter levels, and the required patch changeset. SecurityRisk: none The change only restructures message data and performs no new fetching, execution, deserialization, credential handling, or trust-boundary crossing. TestingStatus: appropriate Regression coverage exercises URL, binary, base64, data-URL, URL-object, and image-file paths plus the v1 marker; focused checks also confirmed ArrayBuffer handling, non-image preservation, and OpenAI downstream translation. VerificationReviewed the complete merge-base diff and relevant LangChain types and provider converters. All 213 Node tests and 213 edge-runtime tests passed, as did package type checking, build, formatting, linting, diff validation, and focused canonical/downstream conversion assertions. Relevant Documentation |
|
🚀 Published in:
|
1 participant
Background
The published @ai-sdk/langchain adapter returned OpenAI-specific image_url blocks for URL and binary images instead of LangChain's canonical image blocks.
Root Cause
convertUserContent hard-coded OpenAI image serialization and constructed multimodal HumanMessages through content rather than contentBlocks. The reproduction confirmed inconsistent image output, and live ChatOpenAI validation showed canonical blocks require LangChain's v1 content marker for downstream translation.
Summary
Converted image inputs and image files to canonical LangChain image blocks and constructed multimodal messages through contentBlocks so LangChain marks and translates them as v1 content. Added a patch changeset.
Testing
Updated convertUserContent and convertModelMessages coverage for URL, binary, data-URL, and file-based images, including the v1 content marker.
End-to-end Validation
pnpm -C packages/langchain build && pnpm -C examples/ai-functions exec tsx -e <assertions>— built the package and confirmed canonical URL/binary image blocks while preserving canonical non-image files.pnpm -C examples/next-langchain exec tsx -e <live ChatOpenAI validation>— GPT-4o accepted the converted JPEG and identified it as a blue-and-yellow macaw.Related Issues
Fixes #11943
Closes #17619
Backport of #17629
Co-authored-by: jessieibarra 58125016+jessieibarra@users.noreply.github.com