Skip to content

v0.2.0 — read the screen, not just act on it

Choose a tag to compare

@bfortunato bfortunato released this 05 Sep 08:29
· 43 commits to main since this release

use-computer now reads the interface as well as acting on it, and every interaction takes the
highest rung it can reach:

  1. Element, through the platform API — the OS presses the button itself. No coordinates.
  2. Element, by coordinate — the tree sees it but cannot operate it, so click its centre.
  3. Pixel, from vision — screenshot, ui-locator, click there. Still the floor, unchanged.

New

  • use-computer windows — what is open, 134 bytes a window. Make this call first.
  • use-computer tree — the accessibility tree of a window: roles, names, states, actions and
    boxes, via AT-SPI on Linux, UI Automation on Windows, AXUIElement on macOS ([tree] extra).
  • Element addressing on every action that targets one — click --role button --name "Invia",
    plus focus, toggle, expand, collapse, select, set-value, show-menu.
  • A whole interaction can now carry no coordinates at all, and end with tree for the new state.

What it costs to read

One window measured through four rounds of work: 19,752 bytes → ~733 tokens. Pruning, a node
budget, clamped text, a compact node, off-screen subtrees counted rather than expanded, and finally
a text rendering carried inside the JSON — stdout is JSON and nothing else is unchanged.

It refuses rather than guesses

An ambiguous selector returns its candidates instead of picking one. A selector that matches
nothing returns a screenshot — the signal to switch to vision. A node with no on-screen position is
never clicked at a coordinate. A --id is checked against the role and name it came with.

Fixed

  • One unresponsive application no longer takes down a whole desktop read.
  • A missing required option exits 2, not 1 with a traceback, on the supported typer floor.