Add support for image inputs

[mattt]

Resolves #19

[Copilot]

Pull Request Overview

This pull request adds multimodal image support to the AnyLanguageModel library, allowing language models to accept and process image inputs alongside text. The implementation adds an ImageSegment type to the Transcript API and updates various language model implementations to handle images according to their capabilities.

Key changes:

• Added ImageSegment type with support for both URL and raw data sources, including platform-specific image encoding helpers
• Updated OpenAI, Anthropic, Gemini, and Ollama models to support multimodal prompts with images
• Added tests for multimodal functionality and error handling for models that don't support images
• Extended LanguageModelSession with convenience methods for responding with images

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
Tests/AnyLanguageModelTests/Shared/Images.swift	New file with shared test fixtures (test image URL and PNG data)
Tests/AnyLanguageModelTests/OpenAILanguageModelTests.swift	Added tests for multimodal prompts with image URL and data
Tests/AnyLanguageModelTests/OllamaLanguageModelTests.swift	Added tests for multimodal prompts with image URL and data
Tests/AnyLanguageModelTests/MockLanguageModelTests.swift	Added comprehensive tests for image handling in MockLanguageModel
Tests/AnyLanguageModelTests/MLXLanguageModelTests.swift	Added tests verifying rejection of image segments
Tests/AnyLanguageModelTests/LlamaLanguageModelTests.swift	Added tests verifying rejection of image segments
Tests/AnyLanguageModelTests/GeminiLanguageModelTests.swift	Added tests for multimodal prompts with image URL and data
Tests/AnyLanguageModelTests/CoreMLLanguageModelTests.swift	Added tests verifying rejection of image segments
Tests/AnyLanguageModelTests/AnthropicLanguageModelTests.swift	Added tests for multimodal prompts with image URL and data
Sources/AnyLanguageModel/Transcript.swift	Added ImageSegment type, encoding helpers, and image case to Segment enum
Sources/AnyLanguageModel/Models/OpenAILanguageModel.swift	Implemented multimodal support by converting segments to blocks with images
Sources/AnyLanguageModel/Models/OllamaLanguageModel.swift	Implemented multimodal support by extracting images from segments
Sources/AnyLanguageModel/Models/MLXLanguageModel.swift	Added error case for unsupported images and handling in tool output
Sources/AnyLanguageModel/Models/LlamaLanguageModel.swift	Added error case for unsupported images
Sources/AnyLanguageModel/Models/GeminiLanguageModel.swift	Implemented multimodal support with inline data and file data parts
Sources/AnyLanguageModel/Models/CoreMLLanguageModel.swift	Added error case for unsupported images and fixed Float cast
Sources/AnyLanguageModel/Models/AnthropicLanguageModel.swift	Implemented multimodal support with image content blocks
Sources/AnyLanguageModel/LanguageModelSession.swift	Added convenience methods for responding with images and reorganized method groups

Comments suppressed due to low confidence (6)

Sources/AnyLanguageModel/Models/MLXLanguageModel.swift:35

• The MLXLanguageModel.respond method does not validate whether the transcript contains image segments, so it will never throw the MLXLanguageModelError.unsupportedFeature error that the tests expect. The model should extract segments using a helper like extractPromptSegments and validate that no image segments are present before processing the prompt.

        public func respond<Content>(
            within session: LanguageModelSession,
            to prompt: Prompt,
            generating type: Content.Type,
            includeSchemaInPrompt: Bool,
            options: GenerationOptions
        ) async throws -> LanguageModelSession.Response<Content> where Content: Generable {

Sources/AnyLanguageModel/Models/MLXLanguageModel.swift:136

• The MLXLanguageModel.streamResponse method does not validate whether the transcript contains image segments, so it will never throw the MLXLanguageModelError.unsupportedFeature error that the tests expect. The model should extract segments and validate that no image segments are present before processing the prompt.

        public func streamResponse<Content>(
            within session: LanguageModelSession,
            to prompt: Prompt,
            generating type: Content.Type,
            includeSchemaInPrompt: Bool,
            options: GenerationOptions
        ) -> sending LanguageModelSession.ResponseStream<Content> where Content: Generable {

Sources/AnyLanguageModel/Models/LlamaLanguageModel.swift:108

• The LlamaLanguageModel.respond method does not validate whether the transcript contains image segments, so it will never throw the LlamaLanguageModelError.unsupportedFeature error that the tests expect. The model should extract segments using a helper like extractPromptSegments and validate that no image segments are present before processing the prompt.

        public func respond<Content>(
            within session: LanguageModelSession,
            to prompt: Prompt,
            generating type: Content.Type,
            includeSchemaInPrompt: Bool,
            options: GenerationOptions
        ) async throws -> LanguageModelSession.Response<Content> where Content: Generable {

Sources/AnyLanguageModel/Models/LlamaLanguageModel.swift:151

• The LlamaLanguageModel.streamResponse method does not validate whether the transcript contains image segments, so it will never throw the LlamaLanguageModelError.unsupportedFeature error that the tests expect. The model should extract segments and validate that no image segments are present before processing the prompt.

        public func streamResponse<Content>(
            within session: LanguageModelSession,
            to prompt: Prompt,
            generating type: Content.Type,
            includeSchemaInPrompt: Bool,
            options: GenerationOptions
        ) -> sending LanguageModelSession.ResponseStream<Content> where Content: Generable {

Sources/AnyLanguageModel/Models/CoreMLLanguageModel.swift:77

• The CoreMLLanguageModel.respond method does not validate whether the transcript contains image segments, so it will never throw the CoreMLLanguageModelError.unsupportedFeature error that the tests expect. The model should extract segments using a helper like extractPromptSegments and validate that no image segments are present before processing the prompt.

        public func respond<Content>(
            within session: LanguageModelSession,
            to prompt: Prompt,
            generating type: Content.Type,
            includeSchemaInPrompt: Bool,
            options: GenerationOptions
        ) async throws -> LanguageModelSession.Response<Content> where Content: Generable {

Sources/AnyLanguageModel/Models/CoreMLLanguageModel.swift:119

• The CoreMLLanguageModel.streamResponse method does not validate whether the transcript contains image segments, so it will never throw the CoreMLLanguageModelError.unsupportedFeature error that the tests expect. The model should extract segments and validate that no image segments are present before processing the prompt.

        public func streamResponse<Content>(
            within session: LanguageModelSession,
            to prompt: Prompt,
            generating type: Content.Type,
            includeSchemaInPrompt: Bool,
            options: GenerationOptions
        ) -> sending LanguageModelSession.ResponseStream<Content> where Content: Generable {

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull Request Overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.