Multimodal (image/document) input: forward A2A file/data parts to the model instead of silently dropping them

[initializ-mk]

Problem

The A2A envelope models multi-part messages with media — but the runtime silently drops everything except text before it reaches the model. So a client can attach a document or image and get a normal-looking response that never actually considered the attachment, with no error and no signal that it was ignored.

What the protocol already supports

Message.Parts[] where Part.Kind is text / data / file (forge-core/a2a/types.go:57-59), and a file part carries media:

type FileContent struct {
    Name     string
    MimeType string   // image/png, application/pdf, ...
    URI      string   // by reference
    Bytes    []byte   // inline (base64 on the wire)
}

POST /tasks/send accepts and stores such a message fine.

Where it's dropped

When the loop projects the A2A message into the LLM request it keeps only text parts:

// forge-core/runtime/loop.go:943
var textParts []string
for _, p := range msg.Parts {
    if p.Kind == a2a.PartKindText && p.Text != "" {
        textParts = append(textParts, p.Text)
    }
}
return llm.ChatMessage{Role: role, Content: strings.Join(textParts, "\n")}

• File and data parts never reach the model.
• llm.ChatMessage.Content is a plain string (forge-core/llm/types.go:17) — no multimodal content array — and there's no image/vision path in the OpenAI/Anthropic providers, so the client type couldn't carry an image even if the loop forwarded it.
• Inbound file bytes aren't persisted anywhere the agent could reach them either (the filesDir / .forge/files is agent output, not inbound), so there's no side-channel workaround.

(For contrast, the reverse direction works: an agent can return a file part — file_create emits PartKindFile, loop.go:804-822.)

Footgun to fix regardless of the full feature

Even before multimodal lands, accepted-but-dropped is a correctness footgun: a client attaching a PDF/image gets a 200 and a plausible answer that ignored the attachment, with nothing telling them. Minimum bar: when compression/projection drops non-text parts on input, either

• return a clear error / 4xx ("this agent does not accept file/image input"), or
• surface a warning field on the response / an audit event (input_parts_dropped with the dropped kinds + mimeTypes),

so the ignored-attachment case is visible instead of silent.

Proposed three-layer plan (full multimodal input)

1. LLM abstraction — give llm.ChatMessage a structured content-parts representation (text blocks + image blocks with {mime_type, bytes|url}), keeping the plain-string Content as the common/back-compat case.
2. Provider layer — translate image/document parts into each provider's native multimodal wire format (OpenAI image_url content parts; Anthropic image source blocks — anthropicContentBlock already exists as the seam). Gate on model capability so a non-vision model gets a clean error, not a malformed request.
3. Runtime projection — in loop.go, project file (and where relevant data) parts into image/document content blocks by mimeType instead of dropping them; decide the policy for unsupported types (reject vs. describe-as-text).

Scope / open questions

• Which media to support first — images (vision) is the obvious first cut; PDFs/documents may need extraction or provider-native document support.
• bytes (inline base64) vs uri (fetch) handling, and how uri interacts with the egress allowlist.
• Size caps for inline bytes (mirror the audit/tracing capture caps posture).
• Capability detection per model/provider so unsupported combos fail loudly.

Context

• Accepted: forge-core/a2a/types.go (Part / FileContent)
• Dropped: forge-core/runtime/loop.go:943 (text-only projection)
• Client type: forge-core/llm/types.go:17 (ChatMessage.Content string)
• Provider seam: forge-core/llm/providers/anthropic.go (anthropicContentBlock), forge-core/llm/providers/responses.go (responsesContentPart)
• Output already works: forge-core/runtime/loop.go:804-822 (file_create -> PartKindFile)