Repository navigation
Handsy 1.0.6 (TestFlight)
Highlights
- Standby in Dynamic Island: between runs the island offers your most used intents and New Task, so a task starts from whatever app you are in.
- Add to a running task: words typed while it runs join its next step.
- Tasks over many items: the agent lists what matches, asks once with how many, then runs up to 1000 steps; repeated taps are done without a model call each time.
- One memory for the app and the CLI (
esp32agent memory, MCPmemory_recall/memory_add). - Optional absolute-mouse firmware (
hardware/flash.sh --abs): no pointer calibration.
Join the TestFlight beta: https://testflight.apple.com/join/1xQHgJcA · Firmware and setup: https://handsy.tech/install
- Calls: Control Center is closed with the Home key alone, and Home flicks up at most once, only when the key did
nothing: the extra flicks folded the Live Captions panel away mid-call. To hang up, the agent closes Live Speech
first (its keyboard covers the End button) and checks that the call screen says the call ended. - Dynamic Island: the step counter stays on one line, a waiting timer replaces the progress bar and the buttons are
small, so nothing is cut off. - App · CLI: one memory on the phone and the computer. Settings › Memory › ⋯ exports
MEMORY.mdand imports another
copy (an entry with the same id or words keeps the newer one; passwords, codes and long numbers never come in).
The CLI keeps the same file:esp32agent memory list | add | rm | search | export | import, and the MCP tools
memory_recallandmemory_add. - Firmware (optional build) · App: an absolute mouse,
hardware/flash.sh --abs(#99, by @chenshaorong0118). With
AssistiveTouch on, a tap lands at its screen fraction (abs X Y): no homing, calibration or aiming. The status
saysabs: 1and the app sends coordinates; a board flashed back answerserr unknownand the app goes back to
the relative mouse. Home, back and the switch to the previous app are fast strokes; drags glide at the relative
mouse's speed. The default firmware is unchanged. - Agent: a task over many items may run up to 1000 steps (others stay at 300); Home shows the step and how many
items are listed instead of "up to 300". Every 10 steps the run saves its listed items, so Continue picks them up. - Agent: in a task over many items, taps that followed the same tap the same way twice (say "…", then Delete,
then OK) are done by the app from the third time on, each found by its words on the screen and checked to change
it; anything missing, or prices and paying on the screen, hands back to the AI. Fewer model calls per item. - Agent: a task over many items or that deletes (delete, clear, unfollow, all, every, or a repeat-until intent)
lists the matching items first and asks once with how many and which, even when Ask first is Payments only; the
first delete-like tap before that is held once and the AI is told to list first. The question shows up to five
of the items collected. - App: Standby in Dynamic Island (Settings, off by default): between runs the island shows your three most used
intents and New Task. An intent tapped there runs on the app you are in; New Task (also a Control Center control)
opens a quick sheet to type a task or pick an intent, and the run goes back to the app you came from first. While
a run goes, the island's bubble button opens the same sheet to add to the task. A task started while another app
is in front works on that screen instead of going to the Home Screen first. - Agent: words added while a run goes (the field under the running task on Home, the quick task sheet, or Siri
and Shortcuts while a run is on) join its next step: the review shows them, every later step sees them, and that
step goes to the planner so the plan follows. - Record: a demonstration is saved as what was meant. Opening an app from the Home Screen, a folder or Search
becomes oneopenstep; a tap is named by the text at its spot on the screen; taps on the on-screen keyboard
are left out (the type step holds the text) and its return key becomes Return; a detour that ends back on the
same screen is marked left out (mark it correct to keep it). - Record: text typed from the instruction's words follows the instruction: after the words change, the AI types
what the instruction says now. - Record: "Working out the steps" shows which part it is on and the time so far. The review opens at step 1, with
Next Unsure to jump to the yellow steps. Review screens are blurred until tapped (Settings › Blur screens in
reviews). - Intents: editing an intent's instruction checks its steps, name and checklist against the new words and proposes
the changes; nothing changes until you tap Apply. - Intents: a run that fails adds one hint from its notes to the checklist (words only, phone numbers masked);
checklist rows can be swiped away. - App: a recorded step that misses is described to the AI in words (its label, or the text near the tapped spot on
the recorded screen), never by its coordinates, and the AI may skip a step the screen or task no longer needs.
Fixed taps landed on the keyboard, a wrong chat or a photo, and the step's raw JSON showed in the status. - App: in an intent with AI steps, a recorded tap that nothing names is skipped when the screen differs, instead of
being tapped at its old spot. - App: a run of an intent that fails or is stopped demotes the recorded steps that missed: a tap nothing names is
removed, other steps become AI steps. Undo on the intent page puts the old steps back. - App: opening a run's review while it is already showing keeps it open instead of closing and presenting it again
(one possible cause of the review closing by itself right after a stopped run). - App: Home's intent list masks phone numbers in the instruction preview.
- App: the free-line run end sends the board firmware as
0.2.5, notv0.2.5. - App: usage stats for a run that could not start say at which startup step it stopped and a fixed error kind
(never the error text or the task); the free-line run end sends the same step. - Agent: a tap on the Dynamic Island (where the run's Live Activity shows) is never sent: one just above a field
under the island moves a few units down into the field, one higher up goes back to the AI with where to tap
instead. Fixed a search field right under the island opening the agent app and leaving the task's app. - Agent: a tap on the Home Screen's Search pill that reads as no change is not pressed again by itself: Spotlight
over a plain wallpaper can look unchanged, and the second press landed on Spotlight's suggestions and opened
another app. Each tap logs where the pointer went from and to. - Agent:
homeflicks up from the home bar when the Home key left the same app on screen and only something that
moves by itself changed (an animated sheet), not only when nothing changed. Anopenwhose Return did nothing
tells the AI to tap the app in Spotlight's results. - Prompt: a goal to look at, find or browse something ends with done once the screen shows a match, with a short
answer, instead of browsing on; never tap the Dynamic Island.