Summary
The Swift SDK's messaging API (Channel.post, AgentClient.post(to:message:), AgentClient.dm(to:message:)) only accepts text. There is no way to attach binary/image data or a file to a message.
Native apps that want to send agents a visual snapshot of their state can't do it cleanly. The Whiteboard macOS app, for example, needs to send the agent a board snapshot = a PNG render + a scene.json scene graph alongside the user's text.
Request
Add attachment support to the Swift SDK send paths, e.g.:
try await agent.dm(
to: "designer",
message: "match the logo color",
attachments: [
.image(data: pngData, filename: "board.png"),
.text(sceneJSON, filename: "scene.json")
]
)
and surface attachments on the inbound/observer side so receiving agents can read them.
Notes / open questions
- How should attachments be transported — inline (base64 in payload), or uploaded to a blob store with a URL/handle in the message (cf.
relayfile)? A size threshold that switches between the two would be ideal.
- Should the TS and Python SDKs get the same surface for parity?
- For headless CLI-backed agents (codex/claude), delivering attachments as readable file paths in the workspace may be the most useful form.
Motivation
Agent perception in Whiteboard = PNG snapshot + JSON scene graph. Without attachment support we either inline base64 (bloats messages) or write files out-of-band and pass paths. First-class attachment support would make agent perception a one-call operation.
Summary
The Swift SDK's messaging API (
Channel.post,AgentClient.post(to:message:),AgentClient.dm(to:message:)) only accepts text. There is no way to attach binary/image data or a file to a message.Native apps that want to send agents a visual snapshot of their state can't do it cleanly. The Whiteboard macOS app, for example, needs to send the agent a board snapshot = a PNG render + a
scene.jsonscene graph alongside the user's text.Request
Add attachment support to the Swift SDK send paths, e.g.:
and surface attachments on the inbound/observer side so receiving agents can read them.
Notes / open questions
relayfile)? A size threshold that switches between the two would be ideal.Motivation
Agent perception in Whiteboard = PNG snapshot + JSON scene graph. Without attachment support we either inline base64 (bloats messages) or write files out-of-band and pass paths. First-class attachment support would make agent perception a one-call operation.