Skip to content

Handsy 1.0.6 (TestFlight)

Choose a tag to compare

@onehumanbeing onehumanbeing released this 13 Nov 17:29
· 8 commits to main since this release
39aa23e

Highlights

  • Standby in Dynamic Island: between runs the island offers your most used intents and New Task, so a task starts from whatever app you are in.
  • Add to a running task: words typed while it runs join its next step.
  • Tasks over many items: the agent lists what matches, asks once with how many, then runs up to 1000 steps; repeated taps are done without a model call each time.
  • One memory for the app and the CLI (esp32agent memory, MCP memory_recall / memory_add).
  • Optional absolute-mouse firmware (hardware/flash.sh --abs): no pointer calibration.

Join the TestFlight beta: https://testflight.apple.com/join/1xQHgJcA · Firmware and setup: https://handsy.tech/install


  • Calls: Control Center is closed with the Home key alone, and Home flicks up at most once, only when the key did
    nothing: the extra flicks folded the Live Captions panel away mid-call. To hang up, the agent closes Live Speech
    first (its keyboard covers the End button) and checks that the call screen says the call ended.
  • Dynamic Island: the step counter stays on one line, a waiting timer replaces the progress bar and the buttons are
    small, so nothing is cut off.
  • App · CLI: one memory on the phone and the computer. Settings › Memory › ⋯ exports MEMORY.md and imports another
    copy (an entry with the same id or words keeps the newer one; passwords, codes and long numbers never come in).
    The CLI keeps the same file: esp32agent memory list | add | rm | search | export | import, and the MCP tools
    memory_recall and memory_add.
  • Firmware (optional build) · App: an absolute mouse, hardware/flash.sh --abs (#99, by @chenshaorong0118). With
    AssistiveTouch on, a tap lands at its screen fraction (abs X Y): no homing, calibration or aiming. The status
    says abs: 1 and the app sends coordinates; a board flashed back answers err unknown and the app goes back to
    the relative mouse. Home, back and the switch to the previous app are fast strokes; drags glide at the relative
    mouse's speed. The default firmware is unchanged.
  • Agent: a task over many items may run up to 1000 steps (others stay at 300); Home shows the step and how many
    items are listed instead of "up to 300". Every 10 steps the run saves its listed items, so Continue picks them up.
  • Agent: in a task over many items, taps that followed the same tap the same way twice (say "…", then Delete,
    then OK) are done by the app from the third time on, each found by its words on the screen and checked to change
    it; anything missing, or prices and paying on the screen, hands back to the AI. Fewer model calls per item.
  • Agent: a task over many items or that deletes (delete, clear, unfollow, all, every, or a repeat-until intent)
    lists the matching items first and asks once with how many and which, even when Ask first is Payments only; the
    first delete-like tap before that is held once and the AI is told to list first. The question shows up to five
    of the items collected.
  • App: Standby in Dynamic Island (Settings, off by default): between runs the island shows your three most used
    intents and New Task. An intent tapped there runs on the app you are in; New Task (also a Control Center control)
    opens a quick sheet to type a task or pick an intent, and the run goes back to the app you came from first. While
    a run goes, the island's bubble button opens the same sheet to add to the task. A task started while another app
    is in front works on that screen instead of going to the Home Screen first.
  • Agent: words added while a run goes (the field under the running task on Home, the quick task sheet, or Siri
    and Shortcuts while a run is on) join its next step: the review shows them, every later step sees them, and that
    step goes to the planner so the plan follows.
  • Record: a demonstration is saved as what was meant. Opening an app from the Home Screen, a folder or Search
    becomes one open step; a tap is named by the text at its spot on the screen; taps on the on-screen keyboard
    are left out (the type step holds the text) and its return key becomes Return; a detour that ends back on the
    same screen is marked left out (mark it correct to keep it).
  • Record: text typed from the instruction's words follows the instruction: after the words change, the AI types
    what the instruction says now.
  • Record: "Working out the steps" shows which part it is on and the time so far. The review opens at step 1, with
    Next Unsure to jump to the yellow steps. Review screens are blurred until tapped (Settings › Blur screens in
    reviews).
  • Intents: editing an intent's instruction checks its steps, name and checklist against the new words and proposes
    the changes; nothing changes until you tap Apply.
  • Intents: a run that fails adds one hint from its notes to the checklist (words only, phone numbers masked);
    checklist rows can be swiped away.
  • App: a recorded step that misses is described to the AI in words (its label, or the text near the tapped spot on
    the recorded screen), never by its coordinates, and the AI may skip a step the screen or task no longer needs.
    Fixed taps landed on the keyboard, a wrong chat or a photo, and the step's raw JSON showed in the status.
  • App: in an intent with AI steps, a recorded tap that nothing names is skipped when the screen differs, instead of
    being tapped at its old spot.
  • App: a run of an intent that fails or is stopped demotes the recorded steps that missed: a tap nothing names is
    removed, other steps become AI steps. Undo on the intent page puts the old steps back.
  • App: opening a run's review while it is already showing keeps it open instead of closing and presenting it again
    (one possible cause of the review closing by itself right after a stopped run).
  • App: Home's intent list masks phone numbers in the instruction preview.
  • App: the free-line run end sends the board firmware as 0.2.5, not v0.2.5.
  • App: usage stats for a run that could not start say at which startup step it stopped and a fixed error kind
    (never the error text or the task); the free-line run end sends the same step.
  • Agent: a tap on the Dynamic Island (where the run's Live Activity shows) is never sent: one just above a field
    under the island moves a few units down into the field, one higher up goes back to the AI with where to tap
    instead. Fixed a search field right under the island opening the agent app and leaving the task's app.
  • Agent: a tap on the Home Screen's Search pill that reads as no change is not pressed again by itself: Spotlight
    over a plain wallpaper can look unchanged, and the second press landed on Spotlight's suggestions and opened
    another app. Each tap logs where the pointer went from and to.
  • Agent: home flicks up from the home bar when the Home key left the same app on screen and only something that
    moves by itself changed (an animated sheet), not only when nothing changed. An open whose Return did nothing
    tells the AI to tap the app in Spotlight's results.
  • Prompt: a goal to look at, find or browse something ends with done once the screen shows a match, with a short
    answer, instead of browsing on; never tap the Dynamic Island.