Magic Pointer — a Google Magic Pointer alternative

In May 2026, Google DeepMind announced Magic Pointer: turning the mouse cursor into an AI interface that understands "this" and "that" through Gemini, by capturing the visual context around the pointer (press coverage, MarkTechPost).

This repo is a homemade implementation of the same concept, running entirely on macOS, powered by Mistral instead of Gemini:

Hold Command, draw a red trail with your mouse to point at an area of the screen while speaking, release: the screenshot (with the trail) and your voice transcript are sent to the Mistral API, and the answer streams into a floating bubble that follows your cursor.

Everything runs locally on the Mac — only the Mistral API calls leave the machine (no telemetry, no proprietary cloud).

How it works

1. Hold Command → a red trail appears, draw over the area you care about while speaking.
2. Release → the screenshot (with the trail) and the audio are sent to Mistral: voxtral-mini-latest for transcription, mistral-medium-3.5 for vision analysis.
3. A floating bubble (native macOS vibrancy blur, rounded corners) appears near the cursor with an iMessage-style typing animation while waiting, then displays the answer streaming as it's generated, with basic markdown rendering (**bold**, `code`).
4. The bubble follows the cursor in real time (60 Hz), stays visible across Spaces and full-screen apps, and dismisses itself automatically a few seconds after the answer finishes.
5. A real keyboard shortcut (Cmd+C, Cmd+Tab, Cmd+click...) while Command is held cancels the capture — only holding Command alone triggers the gesture.

Installation

python3 -m venv venv
./venv/bin/pip install -r requirements.txt
cp .env.example .env
# then edit .env and set MISTRAL_API_KEY=your_key

Required macOS permissions

Grant these (System Settings > Privacy & Security) to the Python binary running the script:

• Accessibility — to detect the global Command hold
• Screen Recording — for the screenshot capture
• Microphone — for voice input

Run

./venv/bin/python main.py

Then: hold Command, draw, speak, release.

Structure

• magic_pointer/hotkey.py — Command-alone hold detection (CGEventTap), cancels on a real shortcut
• magic_pointer/overlay.py — transparent window + red mouse trail
• magic_pointer/audio_capture.py — microphone recording
• magic_pointer/screen_capture.py — full-screen capture
• magic_pointer/mistral_client.py — Mistral API calls (transcription + streaming vision)
• magic_pointer/result_panel.py — floating bubble (vibrancy blur, cursor follow, typing animation, markdown)
• magic_pointer/display.py — background-thread → bubble bridge (thread-safe via AppHelper.callAfter)
• main.py — orchestrator

Known limitations

• Only one screen is handled (the one under the cursor at hold time)
• No automated action execution (click, typing...) — only a displayed answer
• Markdown rendering limited to **bold** and `code`

Inspiration

• Shaping the future of AI interaction by reimagining the mouse pointer — Google DeepMind
• Google's AI-enabled mouse pointer understands 'this' and 'that' — The Register