Repository navigation
Handsy 1.0.1 (TestFlight)
Highlights
The first iPhone app. No computer in the middle: the ESP32 board is the AirPlay receiver and the iPhone's USB mouse and keyboard, and the app on the iPhone is the agent. It plans, looks at the mirrored screen and acts, with a Live Activity, Siri and Shortcuts, and recovers by itself when the board, hotspot or mirroring drops.
Join the TestFlight beta: https://testflight.apple.com/join/1xQHgJcA · Firmware and setup: https://handsy.tech/install
TestFlight build 9. Tagged on the main commit that carries this version.
- No computer: the ESP32 is the AirPlay receiver (ported from UxPlay) with its own hotspot, and talks to a new
iPhone app over BLE. The app is the agent: planner/actor loop, pointer calibration, run replay, Live Activity,
widgets, Siri and Shortcuts, self-healing when the board, hotspot or mirroring drops. - Prompts moved to
prompts/. - Product name: ESP32 Agent.
server/is legacy and no longer needed.- Notebook and memory:
readcopies items off the screen into a run's notebook (scrolled overlaps merged),
writeputs it on the clipboard forkey cmd+v; long-term memory inDocuments/memory/MEMORY.md
(pi-memory layout), only the user's own words and lessons from finished runs, never secrets. - Chat between runs: earlier turns reach the next task; "continue" picks up where a run stopped.
- Intents as scripts:
ai,read,write,manualsteps, a missed step is repaired by the AI alone,
until+maxLoopsrepeat the steps. Recordings proposeai/readsteps instead of dropping unsure ones. - Pause / Continue on the Dynamic Island; Ask me first: payments only (default), also sending/deleting, or never;
steps, time, AI calls and tokens per run. - Fixed (intermittent): the home-bar flick could start above the home bar when the pointer calibration was off
near the bottom edge, so iOS ignored it and the run stopped with "The board isn't controlling this iPhone"
(iPhone 17 Pro Max, iOS 26.3). The flick now pushes the pointer past the bottom edge first. - A model's
key deleteis sent asbackspace. - Fixed: a Wi-Fi video reconnect after the board's GOP buffer filled (a still screen, ~30 min) froze the
screen until Screen Mirroring was restarted. The app now reconnects withhello <token> at <ms>and the board
goes on right after that unit, also past a full GOP (firmware + app; docs/ble-protocol.md). - Fixed: after a still screen, the GOP the board resends on reconnect (over a thousand frames) was turned into
one JPEG per frame; iOS killed the app for CPU. Decoded frames are thinned to one per 0.15 s, newest last.
