Skip to content

v0.3.0

Latest

Choose a tag to compare

@SepyMovasat SepyMovasat released this 08 Jun 01:58
Immutable release. Only release title and notes can be modified.

Added

  • No-vision MCP mode. Setting TOUCHPOINT_MODE=no-vision lets agents
    drive the desktop from a compact structured snapshot() of the active
    window — plus diff_snapshot() and automatic verify flags on actions —
    instead of screenshots. Any model can now control the desktop, including
    local models with no vision capability.
  • Cross-platform window management. New APIs tp.minimize_window(),
    tp.fullscreen_window(), tp.close_window(), tp.move_window(), and
    tp.resize_window() (with matching MCP tools) let agents arrange and
    control application windows on Linux, Windows, and macOS.
  • select_text now works everywhere. Native structured text selection
    is complete across all backends — Linux, Windows, macOS, and web/Electron
    — so agents can select substrings without mouse drags or triple-clicks on
    any platform.
  • Read element text directly. The read_text MCP tool and
    tp.get_text_content() return the full text of an element or container
    (article, document body, terminal) verbatim — no screenshot OCR needed.
  • tp.diagnostics() (and a matching MCP tool) reports the health of
    backends, input, CDP targets, timeouts, and optional dependencies, making
    it easy to confirm a working setup or troubleshoot a misconfigured one.
  • Role-classification sets. INTERACTIVE_ROLES, CONTAINER_ROLES, and
    STRUCTURAL_ROLES are exported for grouping elements by kind (e.g. what an
    agent can act on vs. layout containers).

Changed

  • Broader role and state coverage on Windows and macOS. Both backends
    now recognize many more element kinds (headings, landmarks, form roles,
    labels, figures, notes, meters, split buttons, and more) and additional
    states such as pressed, active, invalid, multi-selectable, and resizable —
    bringing Windows and macOS much closer to parity with Linux.
  • Thread-safe, more reliable automation. Public API calls and complete
    MCP tool workflows now serialize access to shared backend and session
    state, so concurrent or worker-thread usage no longer corrupts state or
    stalls.
  • macOS responsiveness hardening. macOS now uses a configurable
    messaging timeout (ax_messaging_timeout / TOUCHPOINT_AX_MESSAGING_TIMEOUT,
    default 1 second) applied throughout element traversal, so an unresponsive
    app is detected and skipped instead of hanging the whole session.
  • Faster element search on large Windows desktops. Windows searches now
    skip obvious leaf controls, use direct control-type filters for exact role
    queries, and fully read only matched elements, reducing overhead on big
    accessibility trees.
  • tp.configure() is now also a getter — calling it with no arguments
    returns a copy of the current configuration.

Removed

  • The elements MCP tool has been replaced by snapshot(), which returns
    a compact structured tree of the active window (in both vision and no-vision
    modes). The tp.elements() Python API is unchanged.

Fixed

  • CDP windows now support OS-level window management. Minimize, fullscreen,
    close, move, and resize on a browser/Electron page are routed to the
    underlying native OS window (resolved by owning process), with a clear error
    only when no native window can be found. (Window activation was already
    supported.)
  • GTK4 apps are now controllable. Some GTK4 apps expose elements under
    UUID-style paths that were previously misread as malformed, so they appeared
    in listings but every action failed. They now work correctly.
  • Consistent errors for malformed element IDs. All backends now raise a
    clear ValueError when given a structurally invalid element ID, instead of
    some platforms silently returning nothing.
  • Reliable Windows text selection. Text selection reads and selects from a
    single source to avoid offset drift, and falls back to a native message for
    classic edit controls that don't expose the modern text pattern. Multiline
    edit fields are no longer mislabeled as single-line, and ordinary controls
    are no longer falsely marked invalid.
  • wait_for correctness. wait_for(..., wait_for_new=True) now records a
    correct baseline of existing matches (previously it could behave as a no-op),
    mode is validated immediately, and repeated queries are de-duplicated within
    each polling cycle to avoid redundant tree scans.
  • Multi-monitor screenshots. Captures of elements on monitors positioned
    above or to the left of the primary display now expand correctly instead of
    being clipped back to the primary display's origin.

Platform support after 0.3.0

Feature Linux (AT-SPI2) Windows (UIA) macOS (AX) Web / Electron (CDP)
select_text ✅ Full ✅ Full ✅ Full ✅ Full
Window management ✅ X11 + wmctrl ✅ Full ✅ Full 🚧 Routed to OS window