fix(adk): preserve multimodal content fields in rewriteMessage#847
Merged
shentongmartin merged 3 commits intomainfrom Mar 12, 2026
Merged
fix(adk): preserve multimodal content fields in rewriteMessage#847shentongmartin merged 3 commits intomainfrom
shentongmartin merged 3 commits intomainfrom
Conversation
When rewriteMessage rewrites a ChatModelAgent's output as a User message for history context, it now carries over MultiContent, UserInputMultiContent, and converts AssistantGenMultiContent to UserInputMultiContent (text/image/ audio/video parts). Reasoning parts are dropped as they have no user input equivalent. Change-Id: If645454287b3fb5da0634b71b2e2eed7a3692c08
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #847 +/- ##
==========================================
+ Coverage 80.53% 80.60% +0.06%
==========================================
Files 129 129
Lines 13095 13129 +34
==========================================
+ Hits 10546 10582 +36
+ Misses 1749 1747 -2
Partials 800 800 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Change-Id: Ib433159ec54876d295e6c43544f4cae2f3b1789e
Change-Id: I90080a89c96ab34f0620c2cef011ab6f7e270973
hi-pender
approved these changes
Mar 12, 2026
meguminnnnnnnnn
pushed a commit
that referenced
this pull request
Mar 13, 2026
* fix(adk): preserve multimodal content fields in rewriteMessage When rewriteMessage rewrites a ChatModelAgent's output as a User message for history context, it now carries over MultiContent, UserInputMultiContent, and converts AssistantGenMultiContent to UserInputMultiContent (text/image/ audio/video parts). Reasoning parts are dropped as they have no user input equivalent. Change-Id: If645454287b3fb5da0634b71b2e2eed7a3692c08 * fix(adk): use composite literal for slice copy in rewriteMessage Change-Id: Ib433159ec54876d295e6c43544f4cae2f3b1789e * fix(adk): preserve nil slice semantics in rewriteMessage Change-Id: I90080a89c96ab34f0620c2cef011ab6f7e270973
shentongmartin
added a commit
that referenced
this pull request
Mar 17, 2026
* fix(adk): preserve multimodal content fields in rewriteMessage When rewriteMessage rewrites a ChatModelAgent's output as a User message for history context, it now carries over MultiContent, UserInputMultiContent, and converts AssistantGenMultiContent to UserInputMultiContent (text/image/ audio/video parts). Reasoning parts are dropped as they have no user input equivalent. Change-Id: If645454287b3fb5da0634b71b2e2eed7a3692c08 * fix(adk): use composite literal for slice copy in rewriteMessage Change-Id: Ib433159ec54876d295e6c43544f4cae2f3b1789e * fix(adk): preserve nil slice semantics in rewriteMessage Change-Id: I90080a89c96ab34f0620c2cef011ab6f7e270973
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a
ChatModelAgentinside aWorkflowAgentproduces a response, its output message is rewritten as aUser-role history entry for subsequent agents in the workflow. The previous implementation ofrewriteMessagecreated a brand-newschema.UserMessage(text)from only the text content and tool calls, silently discarding all other message fields — including multimodal content (MultiContent,UserInputMultiContent,AssistantGenMultiContent).Summary
rewriteMessagedropped all fields exceptContentandToolCallsMultiContentandUserInputMultiContentas new slices; convertAssistantGenMultiContenttoUserInputMultiContentAssistantGenMultiContent(output parts) incompatible with User-role messageMessageInputPart; dropReasoningparts which have no user-input equivalentappend([]T(nil), src...)to give each rewritten message its own independent sliceKey Insight
Role boundary requires type conversion, not just copy.
AssistantGenMultiContentusesMessageOutputPart(withMessageOutputImage,MessageOutputAudio, etc.), while a User message expectsMessageInputPart(withMessageInputImage,MessageInputAudio, etc.). Both shareMessagePartCommonas their embedded base, so the conversion is lossless for all media types.Reasoningparts have no input-side equivalent and are intentionally dropped — they describe the model's internal thinking process, which is not meaningful as user context.Slice independence matters for history safety.
The rewritten message is inserted into the history that may be retained and replayed across workflow iterations. A direct slice assignment would alias the backing array of the original message, risking cross-message mutation. The fix copies slices with fresh backing arrays while sharing inner pointer fields (
*MessageInputImage, etc.), which are treated as immutable once set.Test
Added
TestRewriteMessageinadk/flow_test.gocovering:MultiContentcopied independently (mutation of copy doesn't affect original)UserInputMultiContentcopied independentlyAssistantGenMultiContentpart types (text, image, audio, video) converted correctly withExtraandMessagePartCommonpreservedReasoningparts droppedAssistantGenMultiContentnot set on rewritten messagerewriteMessageandgenMsgboth reach 100% statement coverage.问题
当
WorkflowAgent中的ChatModelAgent产生响应时,其输出消息会被改写为后续 agent 的历史条目(User角色)。原有的rewriteMessage实现仅从文本内容和工具调用构建新的schema.UserMessage(text),会静默丢弃消息中的所有其他字段,包括多模态内容字段(MultiContent、UserInputMultiContent、AssistantGenMultiContent)。变更概要
rewriteMessage丢弃了除Content和ToolCalls以外的所有字段MultiContent和UserInputMultiContent;将AssistantGenMultiContent转换为UserInputMultiContentAssistantGenMultiContent(输出部分)与 User 角色消息不兼容MessageInputPart;丢弃无用户输入等价物的Reasoningpartappend([]T(nil), src...)为改写后的消息分配独立 slice核心洞察
角色边界需要类型转换,而不仅仅是拷贝。
AssistantGenMultiContent使用MessageOutputPart(含MessageOutputImage、MessageOutputAudio等),而 User 消息期望MessageInputPart(含MessageInputImage、MessageInputAudio等)。两者均以MessagePartCommon作为嵌入基类,因此所有媒体类型的转换是无损的。Reasoningpart 没有输入侧等价物,被有意丢弃——它描述的是模型的内部思考过程,作为用户上下文没有意义。slice 独立性对历史安全性至关重要。
改写后的消息会被插入历史记录,在 workflow 迭代中可能被保留和重放。直接赋值 slice 会与原始消息共享底层数组,存在跨消息数据被修改的风险。修复方案通过拷贝 slice 分配新的底层数组,同时共享内部指针字段(如
*MessageInputImage)——这些字段一旦设置就被视为不可变。