Skip to content

v0.6.0 - Multimodal Image Support

Choose a tag to compare

@slb350 slb350 released this 14 Nov 15:54
· 222 commits to main since this release

🖼️ Multimodal Image Support

Added comprehensive support for sending images alongside text to vision-capable models following the OpenAI Vision API format.

New Features

Image Loading Methods:

  • ImageBlock::from_url(url) - HTTP/HTTPS URLs
  • ImageBlock::from_file_path(path) - Local files with automatic base64 encoding
    • Supports: .jpg, .jpeg, .png, .gif, .webp, .bmp, .svg
    • MIME type automatically inferred from file extension
  • ImageBlock::from_base64(data, mime) - Manual base64 with explicit MIME type

Convenience Helpers:

  • Message::user_with_image(text, url) - Simple text + image
  • Message::user_with_image_detail(text, url, detail) - With detail level control
  • Message::user_with_base64_image(text, data, mime) - From base64 data

New Types:

  • ImageBlock - Represents an image in a message
  • ImageDetail - Control image processing (Low/High/Auto)
  • OpenAIContent - Message content format (Text or Parts)
  • OpenAIContentPart - Content part for multimodal messages

Examples:

  • examples/vision_example.rs - Comprehensive vision API demonstration

🛡️ Defensive Programming Enhancements

Enhanced Base64 Validation:

  • Character set validation (rejects spaces, special chars)
  • Length validation (must be multiple of 4)
  • Padding validation (max 2 '=' characters at end)
  • MIME injection prevention
  • Large data warnings (>10MB)

Enhanced URL Validation:

  • Control character detection
  • Data URI base64 validation
  • Long URL warnings (>2000 chars)
  • Scheme validation (blocks javascript:, file:, etc.)

Debug Logging:

  • Optional image serialization logging
  • URL truncation for privacy
  • Detail level logging

⚠️ Breaking Changes

ToolUseBlock and ToolResultBlock fields now private - Following Rust API Guidelines (C-STRUCT-PRIVATE):

Migration:

// Before:
tool_use.id
tool_use.name
tool_use.input

// After:
tool_use.id()      // Returns &str
tool_use.name()    // Returns &str
tool_use.input()   // Returns &Value

📦 Installation

[dependencies]
open-agent-sdk = "0.6.0"

Related: Closes #2

Full Changelog: v0.5.0...v0.6.0