feat: allow readFile tool to read images for vision-capable models - #569
Conversation
… with mime type validation
🤖 Generated with [Pochi](https://getpochi.com) Co-Authored-By: Pochi <noreply@getpochi.com>
🤖 Generated with [Pochi](https://getpochi.com) Co-Authored-By: Pochi <noreply@getpochi.com>
This PR implements support for reading image files in the readFile tool. When a model supports image input (e.g., Gemini, Anthropic), the readFile tool can now return base64-encoded image data. The implementation: - Updates the readFile tool to return either text or media output - Adds supportedMimeTypes parameter to control which media types are allowed - Updates all relevant integrations to pass supportedMimeTypes through the tool chain - Adds proper MIME type validation for media files - Adds size limit for media files (20MB) Fixes #539 🤖 Generated with [Pochi](https://getpochi.com) Co-Authored-By: Pochi <noreply@getpochi.com>
This change updates the sanitize-message.ts file to properly handle the new "media" type instead of the old "image" type for readFile tool outputs. This is part of the implementation to allow readFile to read images for supported models. Fixes #539 🤖 Generated with [Pochi](https://getpochi.com) Co-Authored-By: Pochi <noreply@getpochi.com>
Updated test assertions to handle the new union type output from readFile tool where content can be either text or media type. Added type guards to ensure type safety when checking text content. 🤖 Generated with [Pochi](https://getpochi.com) Co-Authored-By: Pochi <noreply@getpochi.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #569 +/- ##
===========================
===========================
☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for reading and processing image files in the readFile tool for vision-capable AI models. The changes enable models with vision capabilities (like Google models) to receive images as base64-encoded data with proper MIME type information, extending the tool beyond text-only file reading.
Key Changes:
- Modified
readFiletool to return a union type supporting both text and media content - Added MIME type detection and validation utilities for media files
- Threaded
supportedMimeTypesthrough the entire tool execution pipeline - Updated message sanitization to handle media content appropriately
Reviewed Changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/vscode/src/tools/read-file.ts | Added media file reading logic with MIME type validation |
| packages/vscode/src/tools/test/read-file.test.ts | Added type guards for union return type handling in tests |
| packages/tools/src/read-file.ts | Converted to factory function with union output schema |
| packages/tools/src/index.ts | Introduced CreateToolOptions interface and threaded supportedMimeTypes |
| packages/common/src/tool-utils/media.ts | New utility for reading and validating media files |
| packages/common/src/tool-utils/fs.ts | Exported isPlainText function and added type field to ProcessedContent |
| packages/livekit-cf/src/lib/sanitize-message.ts | Updated to redact media data instead of just text content |
| packages/vendor-pochi/src/vendor.ts | Added supportedMimeTypes for Google models |
| Multiple integration files | Plumbed supportedMimeTypes through execution contexts |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
feat(media): extend media type support with video and audio formats refactor: simplify media output handling in read-file function
…ove error handling
… output type to json
Screenshot
Summary
Implementation Details
isSupportedMediaFileutility function to validate file types based on MIME typesRelated Issue
Fixes #539
Test Plan
🤖 Generated with Pochi