Computer use in a sandboxed desktop for systems with no API

[ldbumble]

Half of the owner's real job lives in systems with no API: a care-records portal, a payroll thick client, a Citrix session, a vendor site that breaks under the DOM driver. Today Taskuary stops at drafts and proposals for those. This is the one thing that would turn "proposed the month-end close" into "did the month-end close."

The idea

A sandboxed desktop the agent operates, not the owner's own screen:

• a Windows VM (or a container with a desktop) cloned from the owner's setup: same apps, a scoped account with only the permissions the job needs, no personal mail, no password manager;
• the general agent drives it with computer use (pixels) only inside that box, and the owner watches a screencast in the pane the way the browser pane streams today;
• take-over button, like the browser pane: a password, a 2FA code or a judgement call hands the mouse back to the human;
• every run leaves a receipt: the screencast, the steps, what changed, and an undo where one exists.

Why a sandbox and not the owner's screen

docs/browser-vs-computer-use.md settled the order: API, then structured UI (DOM / UI Automation tree), then the shell, then pixels, then the human. Pixels are ~6x the tokens per step and fight the owner for their own display. The verdict was "earns its place only inside an isolated desktop nobody is sitting in front of". This discussion is that desktop.

What already exists to build on

• the browser pane (browserview.py): agent-browser screencast to a canvas, needs:browser tag set at task creation, take-over, saved logins
• the general agent and its approval-gated egress (nothing leaves without a proposal)
• playbooks (playbooks.py): the uses: / alone / ask first lines already say which systems a kind of job touches and what an agent may do unattended

Open questions

• Hosting: a local Hyper-V / WSL desktop on the owner's box, or a cloud VM? Licensing of the apps inside it decides more than we do.
• Scope: which jobs get the desktop at all? Probably only those whose playbook names a system with no connector.
• Recording: a run in the sandbox is also the raw material for Record a job on screen, turn it into a playbook #179 (record a job on screen, turn it into a playbook).

Status: roadmap, not scheduled.