Skip to content

feat: allow readFile tool to read images for vision-capable models - #569

Merged
wsxiaoys merged 13 commits into
mainfrom
fix/read-image
Oct 18, 2025
Merged

feat: allow readFile tool to read images for vision-capable models#569
wsxiaoys merged 13 commits into
mainfrom
fix/read-image

Conversation

@zhanba

@zhanba zhanba commented Oct 17, 2025

Copy link
Copy Markdown
Member

Screenshot

image image

Summary

  • Extends the readFile tool to support reading image files for vision-capable AI models
  • Adds MIME type detection and validation for media files
  • Returns images as base64-encoded data with proper media type information
  • Updates sanitize-message to handle media content types correctly
  • Adds comprehensive test coverage for type safety with union return types

Implementation Details

  • Added isSupportedMediaFile utility function to validate file types based on MIME types
  • Modified readFile to return a union type supporting both text and media content
  • Updated message sanitization logic to properly handle media types instead of just image types
  • Added type guards in tests to ensure type safety when checking content

Related Issue

Fixes #539

Test Plan

  • Unit tests pass for readFile with text files
  • Unit tests pass for readFile with image files
  • Type checking works correctly with union return types
  • Pre-push hook tests pass successfully

🤖 Generated with Pochi

zhanba and others added 7 commits October 17, 2025 12:05
🤖 Generated with [Pochi](https://getpochi.com)

Co-Authored-By: Pochi <noreply@getpochi.com>
🤖 Generated with [Pochi](https://getpochi.com)

Co-Authored-By: Pochi <noreply@getpochi.com>
This PR implements support for reading image files in the readFile tool.
When a model supports image input (e.g., Gemini, Anthropic), the readFile
tool can now return base64-encoded image data.

The implementation:
- Updates the readFile tool to return either text or media output
- Adds supportedMimeTypes parameter to control which media types are allowed
- Updates all relevant integrations to pass supportedMimeTypes through the tool chain
- Adds proper MIME type validation for media files
- Adds size limit for media files (20MB)

Fixes #539

🤖 Generated with [Pochi](https://getpochi.com)

Co-Authored-By: Pochi <noreply@getpochi.com>
This change updates the sanitize-message.ts file to properly handle the new "media" type instead of the old "image" type for readFile tool outputs. This is part of the implementation to allow readFile to read images for supported models.

Fixes #539

🤖 Generated with [Pochi](https://getpochi.com)

Co-Authored-By: Pochi <noreply@getpochi.com>
Updated test assertions to handle the new union type output from readFile
tool where content can be either text or media type. Added type guards to
ensure type safety when checking text content.

🤖 Generated with [Pochi](https://getpochi.com)

Co-Authored-By: Pochi <noreply@getpochi.com>
@codecov-commenter

codecov-commenter commented Oct 17, 2025

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (e220fd6) to head (063433f).
⚠️ Report is 720 commits behind head on main.

Additional details and impacted files
@@     Coverage Diff     @@
##   main   #569   +/-   ##
===========================
===========================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@wsxiaoys
wsxiaoys requested a review from Copilot October 17, 2025 18:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for reading and processing image files in the readFile tool for vision-capable AI models. The changes enable models with vision capabilities (like Google models) to receive images as base64-encoded data with proper MIME type information, extending the tool beyond text-only file reading.

Key Changes:

  • Modified readFile tool to return a union type supporting both text and media content
  • Added MIME type detection and validation utilities for media files
  • Threaded supportedMimeTypes through the entire tool execution pipeline
  • Updated message sanitization to handle media content appropriately

Reviewed Changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
packages/vscode/src/tools/read-file.ts Added media file reading logic with MIME type validation
packages/vscode/src/tools/test/read-file.test.ts Added type guards for union return type handling in tests
packages/tools/src/read-file.ts Converted to factory function with union output schema
packages/tools/src/index.ts Introduced CreateToolOptions interface and threaded supportedMimeTypes
packages/common/src/tool-utils/media.ts New utility for reading and validating media files
packages/common/src/tool-utils/fs.ts Exported isPlainText function and added type field to ProcessedContent
packages/livekit-cf/src/lib/sanitize-message.ts Updated to redact media data instead of just text content
packages/vendor-pochi/src/vendor.ts Added supportedMimeTypes for Google models
Multiple integration files Plumbed supportedMimeTypes through execution contexts

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread packages/common/src/tool-utils/media.ts Outdated
Comment thread packages/vscode/src/tools/read-file.ts Outdated
Comment thread packages/cli/src/tools/read-file.ts Outdated
Comment thread packages/tools/src/read-file.ts Outdated
Comment thread packages/vendor-pochi/src/vendor.ts Outdated
feat(media): extend media type support with video and audio formats
refactor: simplify media output handling in read-file function
@zhanba
zhanba requested a review from wsxiaoys October 18, 2025 02:42
@wsxiaoys
wsxiaoys added this pull request to the merge queue Oct 18, 2025
Merged via the queue into main with commit 514b395 Oct 18, 2025
3 checks passed
@wsxiaoys
wsxiaoys deleted the fix/read-image branch October 18, 2025 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(tools): allow readFile to read images for supported models

4 participants