feat: agent workspace isolation — cooperative input, Agent Space, PiP monitor, Windows VDD

[richard-devbot]

Context

The cursorless UI automation PR eliminates cursor/focus conflicts for accessibility-enabled elements. This issue tracks the next layer of agent/user coexistence — workspace-level isolation so the agent and user can genuinely work in parallel.

Backlog items

1. Linux AT-SPI support

Add pyatspi (D-Bus accessibility API) as an optional dependency. Implement cursorless click/type for GNOME/GTK apps on Linux. Wayland needs ydotool for input injection.

2. Cooperative input locking

Detect user mouse/keyboard activity via CGEventTap (macOS) or a low-level WH_MOUSE_LL/WH_KEYBOARD_LL hook (Windows). Queue agent actions while the user is actively typing or clicking. Resume on idle. Prevents conflicts when the cursorless path falls back to coordinate events.

3. Agent Space (macOS Spaces)

Subscribe to NSWorkspaceActiveSpaceDidChangeNotification. When the agent opens an app, force it to Space 2 using CGSMoveWindowsToManagedSpace (private API) or AppleScript. User stays on Space 1 and sees zero agent windows.

4. Picture-in-picture agent monitor

A floating PySide6 or Electron window on the user's screen showing a live view of the agent's active window. User can observe without switching Space.

5. Windows virtual display confinement (WindowsPC-MCP integration)

Port the Parsec Virtual Display Driver layer from https://github.com/ShikeChen01/WindowsPC-MCP into Operator-Use as an optional Windows-only plugin. Creates a dedicated virtual monitor for the agent — spatial isolation on top of the API-level isolation already shipped.

Reference

• Design doc: docs/plans/2026-03-31-cursorless-ui-automation-design.md
• Inspired by: https://github.com/ShikeChen01/WindowsPC-MCP

[m13v]

the cooperative input locking approach is really smart. we built something similar for our desktop automation framework and the CGEventTap/WH_KEYBOARD_LL detection was the right call for knowing when the user is active. one thing we learned the hard way: the idle timeout needs to be longer than you'd expect, maybe 2-3 seconds after last input, because fast typists have natural pauses between words that can look like idle if your threshold is too short.

for the AT-SPI Linux support, pyatspi works but the D-Bus round trips add noticeable latency compared to macOS AXUIElement which is in-process. batching element queries into a single tree walk rather than individual lookups helps a lot. also ydotool on Wayland is solid but needs root or a uinput group membership, which is a friction point for onboarding.

the Agent Space idea using CGSMoveWindowsToManagedSpace is clever. we ended up just hiding our automation target windows from the agent's own screenshots instead (SCContentFilter with excludingWindows), which is simpler but doesn't give the full spatial isolation you're going for.

[m13v]

our cross-platform desktop automation framework that handles the element interaction via accessibility APIs on both macOS (AXUIElement) and Windows (UI Automation), including the element tree walking and coordinate-free clicking: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs

and the MCP server that wraps it as tools for AI agents: https://github.com/mediar-ai/terminator/blob/main/crates/terminator-mcp-agent/src/server.rs