Skip to content

v0.5.0 — Cross-OS Tier 4 (Linux + Windows vision + input)

Choose a tag to compare

View all tags
@a-radwan-20 a-radwan-20 released this 18 Jun 09:12
· 28 commits to main since this release
v0.5.0
53c9074

Cross-OS Tier 4 — Linux + Windows vision + input

Tier 4 (screen capture + synthetic mouse/keyboard) shipped macOS-only in v0.4.5. It now works on Linux and Windows too — all via zero-CGO shell-out to standard tools, so the single static binary is unchanged. The automation stack is now genuinely cross-platform.

Screen capture (screen.capture)

  • macOS — screencapture (unchanged)
  • Linux / FreeBSD — first available of grim (Wayland), gnome-screenshot, spectacle (KDE), scrot, maim, or ImageMagick's import; a clear "install one of …" error when none is present
  • Windows — PowerShell + .NET System.Drawing (CopyFromScreen)

Synthetic input (input.click / input.type / input.key)

  • macOS — osascript / System Events (unchanged)
  • Linux / FreeBSD — xdotool (X11); a clear "install xdotool" error when absent (Wayland needs a compositor-specific tool)
  • Windows — PowerShell: user32 SetCursorPos + mouse_event for clicks, SendKeys for typing and named keys

Security — two-gate model, unchanged, on every OS

Synthetic input still requires both the tool.input.control capability and the explicit AIBUTLER_ENABLE_SYNTHETIC_INPUT=1 enable flag (default off). Each backend escapes injected text for its own tool:

  • macOS — AppleScript literal escaping
  • Linux — ---terminated exec args, no shell (xdotool type)
  • Windows — SendKeys metacharacter brace-escaping, text passed as a positional arg and never interpolated into the script

Coordinates are validated non-negative; newlines in typed text are rejected (use input.key "return").

Internals

internal/desktop refactored to a per-OS dispatch model (desktop.go + new capture.go + input.go). All backends compile on every platform (runtime-dispatched, no build tags), so cross-compilation and clear "unsupported OS" errors both work. A single cross-OS key table maps friendly names (return, tab, arrows, …) to macOS virtual key codes, X11 keysyms, and Windows SendKeys tokens.

Tests & validation

New unit tests cover the SendKeys escaper, key-table completeness (every supported key resolves on all three backends), the Linux capture-tool preference order, and per-OS dispatch. The macOS backends are exercised live; the Linux and Windows backends are unit-tested for command construction / escaping / key mapping, with real on-device behaviour to be validated on those platforms (they shell out to standard tools, so the main risk is tool-presence and edge-case key handling). Full repo go test -race -count=1 ./... passes (130 packages); cross-compiles clean for darwin/linux/windows amd64 + linux/arm64 under CGO_ENABLED=0. No new dependencies.

Full changelog: https://github.com/LumabyteCo/aibutler/blob/main/CHANGELOG.md

Assets 2
Loading