Repository navigation
Buddy 1.13.0: the agent pointer shows what 1.12.0 left out, as far as Windows and macOS tell it
Buddy 1.13.0: the agent pointer shows what 1.12.0 left out, as far as Windows and macOS tell it
1.12.0 listed what it could not show. This version shows those things where Windows or macOS give a sign, and says plainly where they give none.
Windows
- A press without the mouse in an app you had open already. While a followed task has a program at work that presses through Windows (it has UI Automation loaded and shows no window: any program of an AI agent; of a scheduled task a script), the other windows on screen are listened to as well, up to 20. A press, tick, list entry or filled box there is taken for that task, and the trace says (not its own window: taken for its click). Nothing is taken while your own hand was at the keys or the mouse in the last 5 seconds.
- Typing a program does with the keyboard, in every window, also in a command window, in Firefox and in a program that draws everything itself. Windows marks a key a program made. The trace never has which keys. A key within 1 second of your own key or click is left out (a helper that completes what you type).
- A box that is filled in without the keyboard, and round option boxes in Windows Forms and WPF programs.
- A real window that PowerShell or Python shows (a form, a small tool) is followed like any other. Only the black command window itself is not.
- What stays out, because Windows does not tell it: a press without the mouse in an app that is not the task's own while you are at the keys or the mouse yourself. A program that only sends window messages and never loads UI Automation. A program that has a window of its own and presses other apps from there. A scheduled task's compiled program without a window (only scripts count). The very first press of a script started a blink ago. What a program writes into a command window without the keyboard. In a program that draws everything itself only mouse clicks and keyboard typing are shown. Firefox is not on the PC this was built on: presses without the mouse were not measured there, so nothing was built for it. And in a window that is not the task's own Windows never says who did it: an app that changes something by itself while your hands are off can be taken for the task. The trace always says taken for.
Mac (Buddy Lite)
- A program's typing gets the agent pointer (osascript is typing here), only if Buddy is ticked under System Settings > Privacy & Security > Accessibility. macOS lets no app hear keys without that. Buddy never asks for it by itself; the menu has a line that says whether it is on.
- Still not shown on a Mac: a press a program makes without the mouse (through Accessibility). macOS tells no other app about it.
Checked
- Windows: the tests on a desktop that is never shown (563 passed, 3 skipped; one island test needs a real screen and passes on GitHub). Three test windows kept off the screen (classic, Windows Forms, WPF), pressed and filled through UI Automation by a script without a window: the list entry and filled box of each were told, the round boxes of the Windows Forms and WPF windows, and the WPF button. In a test window that was no task's own, 4 of 4 presses made with the hands off were in the trace as taken for the agent, and a press 0.8 s after the user's own key was left out, as the rule says. Next to real work in Outlook, Teams, Excel, PowerPoint and File Explorer, showing nothing: with the first rules 16 lines took the user's own work for the agent's; everything those apps told came within 4.3 s of the user's key or mouse button (332 reports), so the rule became 5 seconds; with it, 12 minutes of work and 91 reports from Outlook gave no wrong line. The self-test of the built app passed.
- Windows, after the release, with the installed 1.13.0 on the PC it was built on (it updated itself; the pointers were switched off, so this is the trace only): a script without a window pressed a test window kept off the screen that was no task's own. With the hands off, the tick box, list entry, button and filled box were in the trace as taken for the agent (4 of 4). Two presses made while the user was at the keys were left out, as the rule says. One key a program pressed (F15) while Outlook was in front was in the trace as the keyboard ... taken for its typing.
- Mac: on GitHub's Macs (Apple Silicon and Intel) with the built app: a real key another program posted (
osascript) was heard with that program on it and the pointer said osascript is typing here. That a key of Buddy itself and a key with no program on it are left alone was checked with made-up keys only.
Not checked
- Windows: a press by an agent in a real app while somebody looks at it (only test windows off the screen were pressed). An app that changes something by itself while the hands are off for 5 s or more was not seen in the runs. A scheduled task's script that presses through the screen: tests only. Firefox.
- Mac: a real keyboard next to a typing program, the menu line with the permission off and on by hand, and a MacBook Pro 2019. Only GitHub's Intel machine ran it.
Windows: Buddy-win64.zip, 181327201 bytes, SHA-256 a0543a5bfc049f5219b9cbcdc028b7dadc16d6a67e0955697b7ea5abb1752af0.
Mac: Buddy-mac.zip is attached by the Mac build of this release.
iPhone stays 1.1.0.