Repository navigation
Buddy 1.9.0: crew hands - agents really click and type on screen, many at once
Buddy 1.9.0: crew hands
Your other AI agents can now really look, click and type on screen through Buddy, each with its own pointer, several at the same time. Until now an agent could only post a point for its pointer to fly to.
- An agent asks with one line:
Buddy.exe --crew "{\"agent\": \"tester\", \"do\": \"click\", \"target\": \"Save\", \"window\": \"Notepad\"}".doislook(lists what is in a window),clickortype. The command waits, prints one JSON line such as{"ok": true, "did": "Clicked Save", "how": "direct"}and ends with 0 (done), 3 (not done, the line says why) or 1 (Buddy is not running).--out FILEwrites the same line to a file. - Many at once: every agent has its own pointer and its own queue. Buttons, tick boxes, tabs, list items and text boxes are pressed through Windows UI Automation, which needs no mouse, so several agents press things in the same moment and your own mouse stays where it is. Up to 8 agents at a time.
- The real mouse is used only where a program offers no direct way, and for a bare
x/ypoint: one click at a time, then the mouse is put back. - Rules: off until you say yes (Buddy asks the first time an agent tries; Settings, tab Buddy (AI), Let those agents click and type on screen). Only what you can see is clicked. Never Buddy's own windows, never a window's Close, Minimize, Maximize or Exit button. A name like Delete, Send, Remove, Reset or Pay is asked about first, in a small box that does not take the keyboard. What an agent types is never shown in its bubble.
- For your agent:
tools/buddy-hands/in the repository holds a skill (SKILL.md) andbuddy_do.ps1, which does the quoting:buddy_do.ps1 -Agent tester -Do click -Window Notepad -Target Save. - OpenCode, Claude Code and Codex do not start clicking by themselves: an agent clicks when it runs that line.
- Checked on a real screen with the built app against a test window: look, three agents clicking three buttons at once (all direct, the mouse did not move), typing, the first-time question, a Delete button with no and with yes, the Close button (refused). From source also: a covered button (left alone) and one
x/yclick with the real mouse. - Not checked: other programs than the test window, typing through the real keyboard, and an AI product driving it by itself.
Windows: Buddy-win64.zip, 181280627 bytes, SHA-256 872369907cd8303b7fa14218cfdd00ead2c8d4d2c06231f5c9caa3cc397e275a.
Mac stays 1.6.0 and iPhone 1.1.0 (no crew there).