Skip to content

v1.1.0

Latest

Choose a tag to compare

@github-actions github-actions released this 23 Jul 23:36
Immutable release. Only release title and notes can be modified.
0423269

Added

  • Ignore regions for visual comparison. glass_diff, glass_wait_for_region,
    glass_wait_stable, and glass_do's settle action accept ignore — window-relative
    rectangles excluded from the comparison — so perpetually animating content (a blinking text
    caret, a clock, a spinner) no longer keeps changed_pct permanently non-zero or prevents a
    settle from ever completing. glass_diff, glass_wait_stable, and glass_wait_for_region each
    report the excluded count as ignored_pixels — so a mask that covers the whole compared area is
    visible rather than hidden behind a hollow settled/matched — and changed_pct is measured
    over the pixels that remain.
  • How-to: measure the verification-loop cost (semantic vs screenshot) —
    docs/how-to/verification-cost.md.
  • Reference: host compatibility — which MCP hosts are verified against glass, and what glass needs
    from any host — docs/reference/host-compatibility.md.
  • How-to: drive a native iOS app in the Simulator — build the bundled
    examples/ios-greeter/ demo app and drive it end to end (launch, read the
    accessibility tree, act, verify from text and by diff) —
    docs/how-to/drive-an-ios-app.md.

Changed

  • Accessibility is on by default at launch. glass_start now enables the accessibility tools
    without an explicit a11y: true — the semantic path (address elements by #id, verify from text,
    no image tokens) works out of the box, matching how glass is meant to be driven. Pass a11y: false
    to skip spawning the private accessibility bus for canvas/pixel-only apps. On a Linux host that
    can't start an accessibility bus (e.g. at-spi2-core isn't installed), the default quietly falls
    back to pixel-only rather than failing the launch; an explicit a11y: true still fails loudly.
    Linux only in effect; other backends already read accessibility ambiently.
  • Server instructions lead with the low-token accessibility path. The guidance an MCP host
    shows the agent now presents semantic addressing (glass_a11y_snapshotglass_click_element /
    glass_set_value, text-only, no image tokens) as the default way to see and drive the UI, with
    screenshots and pixel coordinates as the fallback for canvas/black-box apps — so an agent reaches
    for the cheap path first. No tool behavior changed.
  • Error messages point to the recovery action. Three common failures now name what to do next
    instead of dead-ending: "no active session" tells you to glass_start; the accessibility-
    unsupported error points to the pixel loop (glass_screenshot + glass_click); and an element
    with no clickable geometry suggests glass_scroll_to_element to bring it into view (or locating it
    by screenshot). The remaining failure paths already carried a next step.
  • glass_a11y_snapshot says what to do when the app exposes no elements. When a snapshot comes
    back with only the window root (no addressable elements) — common for an app that doesn't publish an
    accessibility tree — the result now appends a hint to drive the app by pixels (glass_screenshot +
    glass_click), instead of returning a bare root-only outline. Previously only the Linux "no a11y
    bus" path guided the agent; this covers the thin-tree outcome on every backend.

Fixed

  • Linux a11y:true now reaches AccessKit-based apps (egui/winit and other Rust GUI
    toolkits).
    The private accessibility bus glass spawns for a11y:true advertises a screen
    reader by setting org.a11y.Status.ScreenReaderEnabled, the signal AccessKit-based apps gate
    their accessibility-tree publication on. That setting previously failed to take effect (the
    bus's GSettings backend tried to persist it via a D-Bus service glass's isolated bus doesn't
    expose), so such apps stayed invisible to glass_a11y_snapshot and friends even though the
    bus itself was reachable — GTK/Qt apps, which don't gate on the advertisement, were unaffected.
  • GTK4/Qt apps render under containment on the headless Linux display instead of black. When
    glass launches an app in its sandbox on Linux, it now sets software-render env defaults
    (GSK_RENDERER=cairo, QT_X11_NO_MITSHM=1, QT_QUICK_BACKEND=software) so the GPU/shared-memory
    rendering paths the sandbox blocks don't leave the window black. Pass the relevant variable in
    glass_start's env to override.
  • glass_a11y_marks boxes stay inside the window. An element whose accessibility extent
    reaches past the window edge (toolkit a11y geometry can over-report by ~10–20px) had its outline
    drawn off-frame, where it was clipped at the capture edge and read as a rendering glitch. Each
    mark box is now clamped to the window bounds, so an overrunning edge lands on the frame edge. The
    #id and the glass_click_element target (the element center) are unchanged.

Full changelog: v1.0.1...v1.1.0