Repository navigation
v0.2.0 — read the screen, not just act on it
use-computer now reads the interface as well as acting on it, and every interaction takes the
highest rung it can reach:
- Element, through the platform API — the OS presses the button itself. No coordinates.
- Element, by coordinate — the tree sees it but cannot operate it, so click its centre.
- Pixel, from vision — screenshot, ui-locator, click there. Still the floor, unchanged.
New
use-computer windows— what is open, 134 bytes a window. Make this call first.use-computer tree— the accessibility tree of a window: roles, names, states, actions and
boxes, via AT-SPI on Linux, UI Automation on Windows, AXUIElement on macOS ([tree]extra).- Element addressing on every action that targets one —
click --role button --name "Invia",
plusfocus,toggle,expand,collapse,select,set-value,show-menu. - A whole interaction can now carry no coordinates at all, and end with
treefor the new state.
What it costs to read
One window measured through four rounds of work: 19,752 bytes → ~733 tokens. Pruning, a node
budget, clamped text, a compact node, off-screen subtrees counted rather than expanded, and finally
a text rendering carried inside the JSON — stdout is JSON and nothing else is unchanged.
It refuses rather than guesses
An ambiguous selector returns its candidates instead of picking one. A selector that matches
nothing returns a screenshot — the signal to switch to vision. A node with no on-screen position is
never clicked at a coordinate. A --id is checked against the role and name it came with.
Fixed
- One unresponsive application no longer takes down a whole desktop read.
- A missing required option exits
2, not1with a traceback, on the supported typer floor.