Native Vision OCR

Drag a box over anything on your screen and the text lands on your clipboard. It runs on Apple's Vision framework, the same on-device engine behind Safari's Live Text. No cloud, no API key, no Tesseract. macOS only.

---

Why not just screenshot it?

If you live in AI chat windows, you already do this twenty times a day: something on screen has text you want to ask about, so you screenshot it and paste the image in.

That image is expensive. A language model can't see pixels the way you do. It converts your screenshot into tokens, and a screenshot costs a lot of them, whether the model ends up reading two words or two hundred.

OCR flips it around. You send the text itself. The model gets exactly what it needs and nothing it doesn't.

The numbers, on a real example

Here is the sample in this repo (assets/stock_gs200.jpg), a dense newspaper stock table. This is close to the worst case for OCR, because the image is tiny and crammed with text:

	Screenshot	OCR text
What you paste	626 × 580 image	1,236 characters of text
Tokens consumed	~484	~310
Source	width × height ÷ 750 (Anthropic's documented estimate)	~4 chars per token

Even here, where the picture is small and the text is at maximum density, OCR comes out about a third cheaper. The interesting part is what happens in the normal case.

The normal case is not close

Most screenshots you paste are big and contain little text. A paused video frame, a slide, a chunk of a web page. Those get captured at full resolution:

Screenshot	Image tokens	If it holds ~3 lines of text	OCR wins by
1280 × 720 video frame	~1,230	~30 tokens	~40x
1920 × 1080 slide (resized by the model)	~1,540	~150 tokens	~10x
626 × 580 dense table (above)	~484	~310 tokens	~1.6x

The rule of thumb: the more your screenshot is mostly empty space and graphics with a little text, the more OCR saves you. For the things people actually paste into AI all day, that is 10x to 40x fewer tokens.

Tokens are not the only win

• The model reads it right. It no longer has to OCR your compressed pixels in its head. Small or low contrast text in a screenshot gets misread; copied text is exact.
• It is editable and searchable. You can fix a typo, grep it, diff it, quote one line.
• It is faster and cheaper. Fewer tokens in means a quicker reply and a smaller bill, on every message.

When you should still send the image

OCR is for when you want the words. If you want the model to reason about a chart, a diagram, a UI layout, or anything where the picture is the point, send the screenshot. This tool does not replace that. It replaces the other case, the one where you only ever wanted the text.

---

How it works

A browser extension can't call native macOS frameworks from its sandbox, so this uses Chrome's Native Messaging bridge:

drag a box  ──►  extension screenshots the visible tab, crops to your box
                              │  native messaging (stdio, JSON)
                              ▼
                     Swift host  →  VNRecognizeTextRequest (Apple Vision)
                              │
   clipboard ◄── copy ◄── {ok, text}  ──►  native macOS notification

The frame is grabbed with chrome.tabs.captureVisibleTab and cropped, not by reading pixels off a <video> element. That avoids the cross-origin canvas tainting that breaks the naive approach on sites like YouTube, so it works anywhere you can drag.

Setup

1. Build the native host

   ./build.sh

2. Load the extension

   • Open brave://extensions (or chrome://extensions)
   • Turn on Developer mode
   • Load unpacked and pick the extension/ folder
   • Copy the extension ID off its card

3. Register the native host with that ID

   ./install.sh <EXTENSION_ID>

4. Fully quit and reopen Brave (⌘Q, not just the window). Native host manifests are only read on a fresh launch.

Usage

Start a selection one of two ways:

• Click the toolbar button, or
• Right-click anywhere and choose Select area to OCR

The page dims and you get a crosshair. Drag a box over the text you want and release. The dim eases in and out, a soft flash confirms the capture, and a native macOS notification shows the character count with a preview of the copied text. Press Esc to cancel.

Notes and limits

• macOS only. It depends on Apple Vision.
• The first OCR after a Brave launch can take about a second while the Vision model loads. After that it is fast, and the host process exits between calls so idle memory cost is near zero.
• It runs on the top-level page. Videos inside cross-origin iframes are not supported yet.
• Send the image, not OCR, when the picture itself is what you want the model to look at (see above).

Troubleshooting

• Nothing happens / no crosshair appears. The content script injects on demand, so a stale tab is no longer an issue. If it still fails, you are likely on a page where extensions can't run (chrome://, the Web Store, a raw PDF). Try a normal web page.
• "Specified native messaging host not found" or "Access forbidden". The host manifest is missing or its extension ID doesn't match. Re-run ./install.sh <EXTENSION_ID> and restart Brave.
• No notification banner, only an in-page toast. Allow Brave under System Settings → Notifications.

License

See LICENSE.