agent-desktop

A headless desktop for agents that need GUI applications. Each agent can get its own X display, i3 instance, input focus, screenshots, browser profiles, and VNC port.

Sessions run as your user. They can work with your files and applications. GUI isolation prevents agents from stealing each other's focus; it does not isolate filesystem access or concurrent edits.

Architecture

agentctl session create <name> allocates the next free display and VNC port, then starts three systemd user-service instances:

agent-display@work-1.service   Xvfb :99
agent-wm@work-1.service        i3 socket for work-1
agent-vnc@work-1.service       localhost:5999

agent-display@work-2.service   Xvfb :100
agent-wm@work-2.service        i3 socket for work-2
agent-vnc@work-2.service       localhost:6000

Session creation holds a file lock while it allocates the display and port, so concurrent agents cannot claim the same slot. Each session stores its config in ~/.config/agent-desktop/sessions/ and browser data in ~/.local/share/agent-desktop/<name>/.

agentctl passes the selected display and i3 socket to every command. It does not read $DISPLAY or $I3SOCK from your primary desktop.

Install

./install.sh

Run the installer as yourself. It will ask for sudo through your package manager and prompt you for a VNC password. It supports apt, dnf, and pacman. The installer:

1. installs Xvfb, i3, input, capture, JSON, and VNC tools
2. installs agentctl, spy, and the systemd service templates
3. migrates the old single-display installation after confirmation
4. creates and starts a session named default

The configured sessions start with your user session. To run them while you are logged out:

sudo loginctl enable-linger "$USER"

The bundled skill is in agent-desktop/. Install that directory using the method supported by your agent or skill manager. install.sh does not install the skill.

Sessions

Create one session per agent or task:

agentctl session create issue-142
agentctl session create report-run
agentctl session list

Select the session on each desktop command:

agentctl --session issue-142 status
agentctl --session issue-142 exec xterm
agentctl --session issue-142 windows
agentctl --session issue-142 shot

Manage existing sessions with:

agentctl session stop issue-142
agentctl session start issue-142
agentctl session destroy issue-142

stop keeps the config and starts the session again on login. destroy stops the services and removes the session config. It leaves browser data in ~/.local/share/agent-desktop/<name>/.

Commands without --session use default.

Browser profiles

agentctl exec gives Chrome-family browsers and Firefox a profile inside the selected session's data directory. This keeps browser processes and locks from crossing sessions or opening a window on your primary desktop.

You may pass --user-data-dir or --profile yourself, but the path must remain inside the session profile root shown by agentctl --session <name> status.

Chrome DevTools access

To let browser automation attach to the same signed-in Chrome window, launch Chrome through agentctl with a loopback DevTools endpoint. Use the actual Chrome executable in PATH (for example, google-chrome-unstable when that is where the required profile is available):

agentctl --session issue-142 exec google-chrome-unstable \
  --remote-debugging-address=127.0.0.1 \
  --remote-debugging-port=9222 \
  https://example.com

agentctl exec automatically adds a non-default --user-data-dir under the selected session's profile root. Chrome requires that isolation for remote debugging, so do not launch Chrome directly or replace it with a profile from the primary desktop.

Confirm that the endpoint is ready before attaching:

curl --fail --silent http://127.0.0.1:9222/json/version

Then connect the automation client to 127.0.0.1:9222. Choose another unused loopback port when 9222 is already occupied. Keep the DevTools address bound to 127.0.0.1; do not expose an authenticated browser on an external interface.

If the saved login is in a named Chrome profile within the session data, add its directory explicitly, for example --profile-directory='Profile 1'.

Commands

agentctl [--session NAME] status
agentctl [--session NAME] windows
agentctl [--session NAME] shot [id|mark]
agentctl [--session NAME] exec <command...>
agentctl [--session NAME] type <text>
agentctl [--session NAME] key <keys...>
agentctl [--session NAME] focus <id|mark>
agentctl [--session NAME] mark <id> <name>
agentctl [--session NAME] ws <name>
agentctl [--session NAME] msg <i3 command>

Use marks as stable window handles. agentctl windows provides the window IDs needed to create them without dumping the full i3 tree.

Watching a session

Use spy to watch a session:

spy                 # Watch default session (port 5999)
spy issue-142       # Watch session by name
spy 6000            # Watch specific port number

Alternatively, launch vncviewer directly:

agentctl --session issue-142 status
vncviewer localhost:5999

The actual port may differ when another session or process already uses 5999. Each VNC server binds to loopback. Tunnel the reported port when viewing from another machine:

ssh -L 5999:localhost:5999 your-user@this-host

The viewer accepts input, so you can take over when an application needs human attention.

Password entry

Do not place a password in agentctl type; shell history, process arguments, and the agent transcript can retain it. Focus the field, then run this yourself:

agentctl --session issue-142 secret

The command reads without echo and pipes the value to xdotool through stdin.

Design notes

• The i3 config has no keybindings, bar, borders, or title bars.
• One application per workspace avoids occlusion and gives screenshots the full 1920x1080 resolution.
• agentctl checks the JSON result from i3 because i3-msg can exit zero when an IPC command fails.
• Screenshots are downscaled to 1000 pixels wide before an agent reads them.
• Xvfb runs without a physical display, GPU, desktop session, or compositor.

Remove

Destroy each configured session first:

agentctl session list
agentctl session destroy default
agentctl session destroy <another-name>

Then remove the installed runtime files:

rm ~/.config/systemd/user/agent-display@.service
rm ~/.config/systemd/user/agent-wm@.service
rm ~/.config/systemd/user/agent-vnc@.service
rm ~/.config/agent-desktop/i3-agent.conf.tmpl
rm ~/.local/bin/agentctl
rm ~/.local/bin/spy
systemctl --user daemon-reload

The system packages and saved browser profiles remain in place.