Multimodal Content

Multimodal Content

Flutter AI SDK supports multimodal content including images, audio, and documents.

Content Types

sealed class Content {
  // Base class for all content types
}

class TextContent extends Content { }
class ImageContent extends Content { }
class AudioContent extends Content { }
class DocumentContent extends Content { }
class ToolCallContent extends Content { }
class ToolResultContent extends Content { }

Text Content

Basic text content:

const content = TextContent('Hello, world!');

Image Content

From URL

final response = await ai.chatWithContent([
  const TextContent('What is in this image?'),
  const ImageContent.fromUrl(
    'https://example.com/image.jpg',
    detail: ImageDetail.high,  // low, high, or auto
  ),
]);

From Bytes

import 'dart:io';

final bytes = await File('photo.png').readAsBytes();

final response = await ai.chatWithContent([
  const TextContent('Describe this image'),
  ImageContent.fromBytes(
    bytes,
    mimeType: 'image/png',
    detail: ImageDetail.auto,
  ),
]);

From Base64

final base64Data = 'iVBORw0KGgo...'; // Base64 encoded image

final response = await ai.chatWithContent([
  const TextContent('Analyze this chart'),
  ImageContent.fromBase64(
    base64Data,
    mimeType: 'image/png',
  ),
]);

Image Detail Levels

enum ImageDetail {
  low,   // Faster, cheaper, less detail
  high,  // Slower, more expensive, better analysis
  auto,  // Let the model decide
}

Audio Content

⚠️ Audio is supported by OpenAI and Google AI, but not Anthropic.

From Bytes

final audioBytes = await File('audio.mp3').readAsBytes();

final response = await ai.chatWithContent([
  const TextContent('Transcribe this audio'),
  AudioContent.fromBytes(
    audioBytes,
    mimeType: 'audio/mp3',
  ),
]);

From Base64

final response = await ai.chatWithContent([
  const TextContent('What is being said?'),
  AudioContent.fromBase64(
    base64AudioData,
    mimeType: 'audio/wav',
  ),
]);

Supported Audio Formats

Format	MIME Type	OpenAI	Google AI
MP3	audio/mp3	✅	✅
WAV	audio/wav	✅	✅
FLAC	audio/flac	✅	✅
M4A	audio/m4a	✅	✅
WebM	audio/webm	✅	✅

Document Content

For PDFs and other documents:

final pdfBytes = await File('document.pdf').readAsBytes();

final response = await ai.chatWithContent([
  const TextContent('Summarize this document'),
  DocumentContent.fromBytes(
    pdfBytes,
    mimeType: 'application/pdf',
    fileName: 'report.pdf',
  ),
]);

Multiple Content Items

Combine multiple content types:

final response = await ai.chatWithContent([
  const TextContent('Compare these two images:'),
  const ImageContent.fromUrl('https://example.com/image1.jpg'),
  const ImageContent.fromUrl('https://example.com/image2.jpg'),
  const TextContent('Which one shows a beach?'),
]);

Provider Support

Content Type	OpenAI	Anthropic	Google AI
Text	✅	✅	✅
Image (URL)	✅	✅	✅
Image (Bytes)	✅	✅	✅
Audio	✅	❌	✅
Documents	✅	✅	✅

Image Analysis Examples

Describe an Image

final response = await ai.chatWithContent([
  const TextContent('Describe this image in detail'),
  const ImageContent.fromUrl('https://example.com/photo.jpg'),
]);
print(response.text);

OCR / Text Extraction

final response = await ai.chatWithContent([
  const TextContent('Extract all text from this image'),
  ImageContent.fromBytes(screenshotBytes, mimeType: 'image/png'),
]);

Chart Analysis

final response = await ai.chatWithContent([
  const TextContent('Analyze this chart and summarize the trends'),
  const ImageContent.fromUrl(
    'https://example.com/chart.png',
    detail: ImageDetail.high,
  ),
]);

Code Screenshot

final response = await ai.chatWithContent([
  const TextContent('Review this code and suggest improvements'),
  ImageContent.fromBytes(codeScreenshot, mimeType: 'image/png'),
]);

Flutter Widget Integration

Image Picker Example

import 'package:image_picker/image_picker.dart';

class ImageChatWidget extends StatefulWidget {
  @override
  _ImageChatWidgetState createState() => _ImageChatWidgetState();
}

class _ImageChatWidgetState extends State<ImageChatWidget> {
  final ai = FlutterAI(
    provider: AIProvider.openai,
    config: AIConfig(
      apiKey: 'your-key',
      model: 'gpt-5.5',
    ),
  );
  
  Future<void> analyzeImage() async {
    final picker = ImagePicker();
    final image = await picker.pickImage(source: ImageSource.gallery);
    
    if (image != null) {
      final bytes = await image.readAsBytes();
      
      final response = await ai.chatWithContent([
        const TextContent('What is in this image?'),
        ImageContent.fromBytes(bytes, mimeType: 'image/jpeg'),
      ]);
      
      showDialog(
        context: context,
        builder: (_) => AlertDialog(
          title: Text('Analysis'),
          content: Text(response.text),
        ),
      );
    }
  }
  
  @override
  Widget build(BuildContext context) {
    return ElevatedButton(
      onPressed: analyzeImage,
      child: Text('Analyze Image'),
    );
  }
}

Best Practices

1. Choose Appropriate Detail Level

// For quick analysis
ImageContent.fromUrl(url, detail: ImageDetail.low);

// For detailed analysis (OCR, charts)
ImageContent.fromUrl(url, detail: ImageDetail.high);

2. Optimize Image Size

// Resize large images before sending
import 'package:image/image.dart' as img;

Uint8List optimizeImage(Uint8List bytes, {int maxWidth = 1024}) {
  final image = img.decodeImage(bytes)!;
  
  if (image.width > maxWidth) {
    final resized = img.copyResize(image, width: maxWidth);
    return Uint8List.fromList(img.encodePng(resized));
  }
  
  return bytes;
}

3. Handle Large Files

// Check file size before sending
final file = File('large-image.jpg');
final size = await file.length();

if (size > 20 * 1024 * 1024) { // 20MB
  throw Exception('File too large');
}

4. Error Handling

try {
  final response = await ai.chatWithContent([
    const TextContent('Analyze this'),
    ImageContent.fromBytes(bytes, mimeType: 'image/png'),
  ]);
} on AIContentFilterError catch (e) {
  print('Image was filtered: ${e.message}');
} on AIError catch (e) {
  print('Error: ${e.message}');
}

Related

• Messages and Content - Message structure
• Providers - Provider capabilities
• Example-Image-Analysis - Complete example

---

Document Support by Provider

Provider	Base64 (PDF...)	URL
Anthropic	✅	✅
Google AI	✅	✅
OpenAI	✅	⚠️ passed as a text reference
Ollama	❌	❌