v2026.03.04.1 — Desktop GUI Control

Desktop GUI Control — Control Any Application Like a Human

The agent can now see your screen and control any desktop application — clicking buttons, typing text, scrolling, dragging — at pixel coordinates, just like a human sitting at your computer.

Two Modes

• Local mode — control your own machine directly via pyautogui. No VM needed. Set desktop.mode: local and say "open Excel and make me a chart"
• Remote mode — connect to a VM running the OSWorld HTTP server for sandboxed environments and benchmarks. Set desktop.mode: remote with a vm_ip

9 New Tools

Tool	What it does
desktop_connect	Connect to local desktop or remote VM
desktop_screenshot	Capture current screen as PNG
desktop_click	Click at x,y (left/right/double-click)
desktop_type	Type text, press key, or send hotkey
desktop_scroll	Scroll mouse wheel
desktop_drag	Drag from one position to another
desktop_cursor	Move cursor without clicking
desktop_shell	Run shell command on target machine
desktop_file	Download file from target for inspection

How It Works

User: "Open Excel and create a Q1 revenue spreadsheet"

→ desktop_connect(mode="local")     # Connect to this PC
→ desktop_screenshot                # See the desktop
→ desktop_click(52, 1060)           # Click Excel in Dock
→ desktop_screenshot                # Verify Excel opened
→ desktop_click(85, 200)            # Click cell A1
→ desktop_type("Month")             # Type header
→ desktop_type(key="tab")           # Next cell
→ desktop_type("Revenue")           # Type header
→ ...                               # Fill data, add chart
→ desktop_type(hotkey=["cmd", "s"]) # Save

Not just browsers. Any app — Excel, Photoshop, Terminal, Finder, anything with a GUI.

Architecture

• core/desktop_controller.py — BaseDesktopController ABC with LocalDesktopController (pyautogui) and DesktopController (async HTTP to VM)
• tools/desktop/tools.py — 9 tool classes extending BaseTool
• Full integration: config, planner prompt, agent injection, feature badge, setup.sh

Also in this release

• Agent Commune — social network for AI agents (7 tools)
• Web deployment — deploy to Vercel/Railway + Supabase databases (3 tools)
• Agent organization — self-cloning specialist agents (5 tools)
• Gateway session fix — fixed UNIQUE constraint race on concurrent CLI+Telegram connects

Quick Start

# config.yaml
desktop:
  enabled: true
  mode: local    # or remote + vm_ip

uv pip install -e '.[desktop]'   # install pyautogui

Then tell the agent: "Connect to my desktop and take a screenshot"

---

Full Changelog: v2026.02.28.1...v2026.03.04.1