[Discuss][API] Multimodal content parts in ChatMessage

[Zhuoxi2000]

Follow-up to the 0.4 planning discussion in #862. I volunteered for the “Support invoking multimodal models” Must item there, and @wenjin272 confirmed no one is working on it yet.

The proposal is to add first-class typed content parts to ChatMessage in both Java and Python, while keeping plain-string content fully backward compatible.

Motivation

Today ChatMessage.content is a plain string, even though the chat-model SDKs we already integrate with support multimodal input natively.

There are also two concrete problems in the current code:

1. We already have provider-specific workarounds. For example, Python Anthropic carries native content blocks through extra_args["anthropic_content_blocks"]. This works, but it is untyped and provider-specific.

2. MCP tool results can already contain images, but ChatModelAction eventually stringifies the result before sending it back to the model. So an image currently becomes the string representation of a map containing base64 data rather than actual image content.

A first-class content-part API would give us one framework-level representation for these cases.

Proposed API

Add a flat ContentPart type in both languages with three initial variants:

• text
• image
• document

Each part would carry only the fields it needs, e.g. text, mime_type, and either data or url.

I’d keep it flat rather than introducing a nested provider-style source object. Besides being simpler across providers, this also works better with the current Event Log depth limits.

ChatMessage would keep its existing content: String and gain contentParts / content_parts, defaulting to empty.

The compatibility rule would be:

• no parts → content behaves exactly as it does today
• parts present → parts are authoritative, while content remains a plain-text projection of the text parts for existing consumers

I considered making content itself a String | List<ContentPart> union, but that would be much more disruptive across Java callers, Jackson mapping, and the pemja bridge. An additive field seems safer for 0.4.

This also requires one small framework fix: image-only messages must not be dropped by the current “empty text content” filtering in the prompt/message merge logic.

Cross-language / serialization

content_parts would be additive in the serialized Event Log format, with old records continuing to deserialize normally.

The pemja bridge needs explicit support because those conversions copy fields directly rather than going through JSON.

I’d leave multimodal YAML prompts out of scope for 0.4. Existing text YAML stays unchanged, and YAML parts: can be added separately once the API settles.

Event Log and state

At STANDARD Event Log level, inline base64 is already bounded by the existing string truncation limit, so this does not make logs unbounded by default.

There is still a state-size concern for long agent/tool loops, since inline media may be carried through sensory memory. For 0.4, my preference would be to recommend URL-backed parts for large media and leave a first-class blob/reference type for follow-up work.

Proposed 0.4 scope

Phase 1 — framework

• ContentPart + ChatMessage support in Java and Python
• Event Log / cross-language serialization
• pemja bridge
• image-only message filtering fix
• round-trip and snapshot tests

Phase 2 — initial providers

Start with OpenAI Chat Completions and Ollama.

OpenAI is a good first target because the shared conversion path also covers Azure OpenAI and vLLM. Ollama already has direct image support in both SDKs.

I’d start with USER-role multimodal input, which covers the primary invocation use case.

Follow-ups

Anthropic, Gemini, Bedrock, Responses API, Tongyi, MCP image tool results, YAML parts, and richer media-reference support can follow incrementally.

Open questions

1. Should Event Log store a truncated base64 prefix, or replace media payloads with metadata such as <image/png, 123 KB>? In particular, should VERBOSE ever record full media bytes?
2. Is recommending URL-backed media enough for 0.4, or should we introduce a reference/blob type now?
3. For existing extra_args escape hatches, my inclination is to gradually subsume provider-specific ones like anthropic_content_blocks, while leaving generic extra_args available for other provider-specific options.
4. Naming: content_parts or simply parts?

If this direction looks reasonable, I’ll open a tracking issue with Phase 1 / Phase 2 subtasks and start with the framework work.

[wenjin272]

Thanks for bringing this up. I’m tied up with other things right now, but I’ll try to check it out this week.

[wenjin272]

Thanks for proposing this. I agree that supporting multimodal messages is necessary for Flink Agents.

At the moment, however, I feel the proposal is still too high-level to evaluate as an API change. It would be helpful to extend it with two concrete parts:

1. The public data model

Please provide the proposed Java and Python structures, including their serialized representation. For example, one possible shape could be:

ChatMessage
  - role
  - blocks: List<ContentBlock>

ContentBlock
  - TextBlock
  - ImageBlock
  - AudioBlock
  - VideoBlock
  - DocumentBlock / FileBlock

The proposal should also define:

• How binary content is represented, such as URL, URI, bytes, or Base64.
• How block types are identified during serialization and cross-language communication.
• Whether mixed blocks preserve their original order.
• Whether tool results can contain the same content block types.

Since Flink Agents is currently in Beta and we do not need to preserve this API for compatibility, I would prefer replacing content: String with the block list instead of keeping both. Having two representations would introduce two sources of truth. Convenience APIs can still be provided for creating text-only messages or extracting their text content.

2. Integration with model providers

The proposal should explain how these blocks flow into a model provider, for example:

ChatMessage blocks
    -> ChatModelConnection/provider adapter
    -> provider-native request content

Each provider uses a different representation for multimodal input. Therefore, it would be useful to clarify:

• Whether each ChatModelConnection implementation is responsible for converting ContentBlocks into its provider-native format.
• How unsupported block types are handled: explicit validation/error, capability declaration, or fallback conversion.
• Whether provider responses and tool results can also return multimodal blocks.
• How this works consistently across the Java and Python implementations.

Frameworks such as LlamaIndex and AgentScope demonstrate different valid approaches: LlamaIndex exposes modality-specific block types, while AgentScope uses a more generic DataBlock routed by MIME type. I do not think Flink Agents must copy either design, but the proposal should make an explicit choice and provide at least one end-to-end example, such as a text-and-image user message being converted into an OpenAI, Anthropic, or Gemini request.

[wenjin272]

Here are my thoughts on the four open questions.

1. Event Log representation

I would prefer recording media metadata rather than a truncated Base64 prefix. A Base64 prefix has little diagnostic value, but it can still expose part of the original content.

For example, the Event Log could contain:

{
  "type": "image",
  "mime_type": "image/png",
  "source_type": "inline",
  "size_bytes": 126432,
  "sha256": "...",
  "data": "<omitted>"
}

For URL-backed content, we could record a sanitized URI or reference identifier, avoiding signed query parameters where appropriate.

I also do not think VERBOSE should automatically record complete media bytes. Binary content can be large and sensitive, and an Event Log should not implicitly become media storage. If preserving the complete payload is needed in the future, it would be better to introduce a separate, explicitly enabled artifact-capture mechanism.

2. URL-backed content versus a Blob/Reference type

For 0.4, I think supporting inline data and externally managed URLs is sufficient. I would not introduce a Flink Agents-managed Blob Storage service as part of this change, since that would also require decisions about storage backends, credentials, cleanup, lifetime, and recovery.

However, the data model should leave room for adding a managed reference later. For example, an image block could initially allow mutually exclusive data and url fields, together with mime_type and optional metadata such as size, checksum, and name. A blob_id or another reference form could then be added later without redesigning the content-block hierarchy.

The documentation should also make it clear that URL-backed content is externally managed. The user is responsible for its accessibility and lifetime. URLs may expire, may not be accessible to the model provider, and may no longer be valid after recovery from a checkpoint. Therefore, recommending URLs reduces state size, but it does not provide durable media storage.

3. Existing extra_args escape hatches

I agree with gradually replacing provider-specific content workarounds such as anthropic_content_blocks once the standard content-block API can represent the same information. Multimodal message content should have one canonical representation and should not be duplicated in both content blocks and extra_args.

Generic extensibility is still useful, so I would not remove extra_args entirely. However, its current name and responsibilities are somewhat ambiguous: it contains message metadata, framework protocol fields, provider response information, and provider-specific extensions. If its intended long-term role is auxiliary message metadata, metadata may be a clearer name, while provider-specific request fields could use a separate mechanism.

Some stable fields currently stored there, such as the tool-call ID and token usage, may also deserve dedicated typed fields. Usage may even belong to a response-level object rather than ChatMessage. This is orthogonal to the multimodal proposal and does not need to block it, but it may be worth tracking separately.

4. Naming

Since Flink Agents is still in Beta and backward compatibility is not required for this API, I would prefer replacing content: String rather than adding a second authoritative representation.

My preferred shape would be:

ChatMessage
  - role
  - blocks: List<ContentBlock>
  - toolCalls
  - metadata

A text-only message would simply contain one TextBlock. Convenience factory methods can preserve the simple text-message experience without storing a separate string field.

Therefore, I would prefer blocks with ContentBlock over either content_parts or parts. If the existing string field must be retained, then content_parts is clearer than the more generic parts, but I think a single canonical blocks field would produce a cleaner API.