Skip to content

v0.4.5 — Tier 4 vision + input (5-tier framework complete)

Choose a tag to compare

View all tags
@a-radwan-20 a-radwan-20 released this 15 Jun 16:31
· 32 commits to main since this release
v0.4.5
3d9fe8f

Tier 4 — vision + input primitives (macOS, zero-CGO)

The last-resort automation tier: screen capture and synthetic mouse/keyboard control, for when no Tier 0–3 path (native API, MCP, scripting, accessibility) can do the job. This completes the 5-tier automation framework.

What's new

internal/desktop adds a Controller and four tools:

  • screen.capture — full-screen screenshot, returned as a base64 PNG data URI. Gated by tool.screen.capture.
  • input.click / input.type / input.key — synthetic pointer click at coordinates, text entry, and named special keys (return, tab, escape, arrows, …). Gated by tool.input.control.

Zero-CGO, shell-out implementation

macOS only in this revision: screencapture for the screenshot, osascript / System Events for mouse + keyboard — all built-in OS tools, no third-party install, no CGO. The single static binary is preserved. Linux (scrot/ImageMagick + xdotool) and Windows (.NET + SendKeys) are scoped follow-ups; the tools return a clear macOS-only error elsewhere.

Security — the highest-risk capability, double-gated

Synthetic input can drive any application with no per-app scoping, so it carries two independent gates:

  1. The tool.input.control capability (opt-in grant), AND
  2. An explicit enable flag (Controller.EnableInput, default OFF), wired in the CLI to AIBUTLER_ENABLE_SYNTHETIC_INPUT=1.

Input stays dead even if the capability is granted, until an operator deliberately turns it on. input.type escapes text for the AppleScript literal and rejects embedded newlines (use input.key "return"); input.click rejects negative coordinates. Screen capture (read-only) uses the separate, lower-risk tool.screen.capture gate.

Tests

7 tests: input-disabled-by-default (both gates), the macOS-only OS gate for capture + input, unknown-key rejection, newline rejection, tool registration, and registry-level denial while input is disabled. Full repo go test -race -count=1 ./... passes across all 130 packages; cross-compiles clean for linux/arm64 and windows/amd64 under CGO_ENABLED=0. No new dependencies.

Milestone — full 5-tier automation framework

With Tier 4 landed, the complete stack is in place: Tier 0 native API · Tier 1 MCP · Tier 2 native scripting (AppleScript/D-Bus/PowerShell/Shortcuts) · Tier 3 accessibility · Tier 4 vision+input — preferring the cheapest, most deterministic tier and falling back to vision-driven input only as a last resort.

Full changelog: https://github.com/LumabyteCo/aibutler/blob/main/CHANGELOG.md

Assets 2
Loading