Skip to content

tine 0.1.0

Latest
Latest

Choose a tag to compare

View all tags
@smythp smythp released this 15 Apr 16:56
· 33 commits to master since this release
v0.1.0
a7af914

tine 0.1.0

First public release of tine — a CLI plus bundled GNOME Shell extension that lets an AI agent drive a GNOME Wayland desktop. AT-SPI2 accessibility reads, labeled-grid vision fallback, OCR text refs (optional), and kernel-level input via /dev/uinput — no Wayland portal consent dialogs.

What's in it

  • AT-SPI2 accessibility tree walker. tine tree, tine tree --app <name>. Assigns short refs (ref_1, ref_2, …) to interactive elements and caches them between calls.
  • Screenshot capture. tine screenshot via the bundled GNOME Shell extension — no Screencast portal, no consent dialog.
    • --annotate overlays Set-of-Mark boxes on known a11y elements.
    • --grid overlays a labeled coordinate grid (A1..Z26) for sparse-tree apps.
    • --ocr runs RapidOCR and adds ref_tN entries for detected text regions (needs the [ocr] extra).
  • Clicking. tine click ref_3 (a11y), tine click ref_t3 (OCR text), tine click B3 (grid cell), tine click 450,320 (pixels), plus tine target for a 3×3 sub-grid refinement crosshair and tine activate ref_3 to invoke the AT-SPI2 action directly (no mouse, no coordinates).
  • Input injection. tine type "text", tine key ctrl+c via python-evdev + /dev/uinput. Kernel-level, no portals.
  • Windows. tine windows lists open windows (title, wm_class, app_id, geometry, focus) via the extension. tine focus Firefox raises and focuses by title match.
  • Describe. tine describe combines screenshot + a11y tree into one payload — the standard "what's on screen?" for an agent turn. --ocr adds OCR text refs.
  • Persistent input daemon. tine inputd start keeps uinput devices warm across commands for an ~8× speedup per action.
  • Bundled GNOME Shell extension. extension/ocr-screenshot@local/ — exposes a D-Bus interface for screenshots, window listing, window focus, and a GLSL invert-colors effect. Apache-2.0, ~280 lines of JS.

Install

# Debian / Ubuntu
sudo apt install python3 python3-gi gir1.2-atspi-2.0 at-spi2-core python3-pip python3-venv

# Create venv with system site-packages so distro's python3-gi is visible
python3 -m venv --system-site-packages ~/.local/share/tine-venv
~/.local/share/tine-venv/bin/pip install tine-cli
ln -sf ~/.local/share/tine-venv/bin/tine ~/.local/bin/tine

Then the one-time uinput setup (udev rule + input group) and the GNOME Shell extension install. See README for the details.

Package name on PyPI is tine-cli (the name tine was already taken); the installed command is tine.

Known limits

  • GNOME Shell 49 Wayland is the only tested target. 47+ should work with shell-version tweaks. KDE / Hyprland / Sway are unsupported — the input half is compositor-agnostic but the bundled extension is GNOME-specific.
  • Linux only. No macOS or Windows plans.
  • /dev/uinput access requires a one-time udev rule — no way around this without portal consent.
  • AT-SPI2 is sparse in Chrome, Electron, and games. Use --grid or --ocr as the fallback.

License

Apache License 2.0.

Assets 2
Loading