vm-use

Computer-use style control of libvirt/QEMU guests, driven from the terminal. vmctl gives an agent (or you) eyes and hands on a VM — screenshots in, pointer and keyboard events out, plus headless command execution — with nothing listening inside the guest.

vmctl shot   windows          # capture the screen -> PNG on disk
vmctl click  windows 640 384  # click where you saw the button
vmctl type   windows "hello"
vmctl exec   ubuntu -- ls /home

Ships with a Claude Code skill (skill/SKILL.md) that teaches an agent the observe/act loop, but vmctl is a plain CLI — it works from any agent harness, a shell script, or your own hands.

How it works

Layer	Mechanism	Available
Vision	virDomain.screenshot() → PNG	Always, including UEFI/boot/login screens
Pointer	QMP input-send-event, type: abs	Needs a USB tablet in the domain
Keyboard	virsh send-key for chords, QMP for typing	Always
Headless exec	qemu-guest-agent guest-exec	After the agent is installed

It all works above the guest OS, through the hypervisor. No SSH, no WinRM, no agent listening on a port, no network path to the guest required. Vision and input need nothing at all installed inside the VM — they work at the UEFI splash and the login screen, before an OS is even up.

---

Setup

Six steps, each with a check so you know it worked before moving on. Budget about fifteen minutes, most of it waiting on the guest-agent install.

1. Host packages and permissions

You need libvirt's Python bindings, Pillow, and membership of the libvirt and kvm groups. That group membership is what grants access to the libvirt socket — there is no polkit rule to write.

# Debian / Ubuntu
sudo apt install python3-libvirt python3-pil

# Fedora / RHEL
sudo dnf install python3-libvirt python3-pillow

# Arch
sudo pacman -S libvirt-python python-pillow

sudo usermod -aG libvirt,kvm "$USER"

Group changes only apply to new logins — log out and back in, or start a new session with exec su - "$USER".

Check: this must list your domains without sudo.

virsh --connect qemu:///system list --all

If it says failed to connect to the hypervisor, your group membership has not taken effect yet. Confirm with id -nG | tr ' ' '\n' | grep -E 'libvirt|kvm'.

Using a session-scope daemon (qemu:///session) instead? That works too — set libvirt_uri in the config in step 4.

2. Give the guest a tablet and an agent channel

Two devices in the domain XML matter. Shut the guest down, then virsh edit <domain> and make sure both are present inside <devices>:

<!-- Absolute pointing. THIS ONE IS REQUIRED for clicking to work at all.
     Without it QEMU only has a relative mouse, and "click at (640, 384)"
     cannot be expressed -- see Design notes. -->
<input type='tablet' bus='usb'/>

<!-- Headless exec, put-file and get-file. Optional: vision and input work
     without it, but you lose `vmctl exec`. -->
<channel type='unix'>
  <target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>

Boot the guest again, and set its display resolution to something you are happy to read screenshots at — 1920×1080 is the sane default. Matching the guest resolution to the screenshot size keeps framebuffer pixels equal to image pixels equal to the coordinates a model reads off the image. vmctl will downscale if it must and will tell you when it does, but that should never be the normal case.

Check:

virsh --connect qemu:///system dumpxml <domain> | grep -E "tablet|guest_agent"

Both lines should appear.

3. Install vm-use

git clone https://github.com/Jackson-Brooks/vm-use.git
cd vm-use
./install.sh

install.sh is two symlinks and a config copy — nothing it does needs root, and you can do it by hand instead:

mkdir -p ~/.local/bin ~/.claude/skills/vm-use ~/.config/vm-use
ln -sfn "$PWD/vmctl" ~/.local/bin/vmctl                         # vmctl on PATH
ln -sfn "$PWD/skill/SKILL.md" ~/.claude/skills/vm-use/SKILL.md  # optional: let
cp vms.example.toml ~/.config/vm-use/vms.toml                   # Claude Code
                                                                # find the skill

It will not overwrite an existing vms.toml.

Symlinks rather than copies, so git pull updates the tool and the skill together.

Check: vmctl --help prints the command list. If the shell cannot find it, ~/.local/bin is not on your PATH.

4. Describe your guests

vmctl does not scan libvirt — you tell it which domains you care about and what you know about them. install.sh already copied the template; now edit it:

$EDITOR ~/.config/vm-use/vms.toml

A minimal entry needs two facts:

[vm.windows]
libvirt_name = "win11-work"   # the domain name `virsh list` shows
platform     = "windows"      # windows | linux | a distro name

The table key (windows) is the short name you type. Both it and the libvirt_name resolve, so vmctl shot windows and vmctl shot win11-work are the same command. See Config reference for the optional keys — and do fill in notes, which is the highest-value one.

Check:

vmctl list

Every guest you defined should appear with its power state. NO SUCH DOMAIN means libvirt_name does not match what virsh list --all shows.

5. Install the guest agent (optional but worth it)

Skip this and everything still works except exec, put-file and get-file — which is to say, you lose the fast path and are left with the slow GUI one. The virtio channel from step 2 must already be in the XML, or the daemon installs, finds no device, and sits idle.

Linux guest:

sudo apt install -y qemu-guest-agent          # or dnf/pacman
sudo systemctl enable --now qemu-guest-agent

Windows guest needs the vioserial driver, which ships with the virtio-win guest tools. On the host, attach the ISO to a spare CD-ROM device:

curl -L -o virtio-win.iso \
  https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso

virsh --connect qemu:///system change-media <domain> sdc "$PWD/virtio-win.iso" \
  --insert --live

Then inside the guest, one double-click: D:\virtio-win-guest-tools.exe. It installs vioserial, the rest of the virtio drivers, and qemu-ga. Eject when done:

virsh --connect qemu:///system change-media <domain> sdc --eject --live

Use the stable-virtio build only. Those are WHQL-signed and install under Secure Boot; latest-virtio is test-signed and would need testsigning on, which Secure Boot forbids.

Check:

vmctl doctor <vm>

Look for guest-exec: enabled. not responding means the daemon is not running or the channel is missing from the XML.

6. Drive it

vmctl doctor is the command to run first, every time — it reports power state, framebuffer size, agent availability, and, most importantly, who currently holds the console, asked of the guest rather than guessed.

vmctl doctor windows      # start here, always
vmctl shot   windows      # prints a PNG path -- open it
vmctl click  windows 640 384
vmctl shot   windows      # confirm it did what you expected

That last line is not optional. A zero exit from click means QEMU accepted the event, not that a button was pressed. The next screenshot is the only evidence. vmctl enforces part of this itself: it refuses to click against a screenshot older than 120 seconds, or one taken before input you have since sent.

If your first screenshot is a lock screen or a login greeter, that is expected — see The console is not the RDP desktop.

---

Config reference

vmctl looks for its guest registry in this order, first hit wins:

1. $VM_USE_CONFIG — explicit path
2. $XDG_CONFIG_HOME/vm-use/vms.toml (default ~/.config/vm-use/vms.toml)
3. vms.toml next to vmctl in the repo

A .json file with the same structure is accepted at any of those paths, for hosts on Python older than 3.11 (tomllib is stdlib from 3.11).

The config lives outside the repo on purpose: it holds host names and addresses, and this way git pull never fights your guest list.

Each guest is a [vm.<key>] table:

Key	Required	Meaning
libvirt_name	yes	Domain name as virsh list --all shows it
platform	yes	windows, linux, or a distro name like ubuntu. Only the Windows/non-Windows split changes behaviour; the string is kept verbatim for anything else reading this file
name		Human label in output. Defaults to the key
description		Free text, unused by vmctl
host, port		Given together, doctor probes them as a readiness check (e.g. RDP on 3389). Omit both to skip it
console_layout		Keyboard layout of the console session. Defaults to us; vmctl type refuses anything else, because it emits US-QWERTY scancodes with no translation
target_resolution		e.g. [1920, 1080]. doctor flags a mismatch
exec_shell		Argv prefix for vmctl exec. Defaults to PowerShell on Windows, ["/bin/bash", "-lc"] elsewhere
notes		Worth filling in — see below

Optional globals at the top level: libvirt_uri (default qemu:///system), virsh_path (default /usr/bin/virsh), and portal_url, quoted in the error when a guest is powered off. Unrecognised keys pass through untouched, so other tools can share the file.

notes is the one to fill in

vmctl doctor prints a guest's notes verbatim, and the bundled skill instructs the agent to treat them as authoritative for that guest. This is where per-machine truth belongs — whether autologin is on, which drives are visible to exec, that one app takes 40 seconds to launch, which button someone must never click. An agent runs doctor first anyway, so the knowledge arrives exactly when it is relevant, instead of rotting in a prompt somewhere.

notes = """
Autologin is on, so a fresh boot lands on the desktop. An active RDP session
moves the session off the console and shows the lock screen -- agent and human
are mutually exclusive. Reclaim with `vmctl console-claim windows`, but ASK
FIRST: it disconnects them.
"""

---

Command reference

vmctl list                              configured guests and their power state
vmctl doctor <vm>                       state, agent, framebuffer, console owner, notes
vmctl console-claim <vm>                move a disconnected session to the console (Windows)
vmctl shot   <vm> [--out PATH]          capture PNG; prints path + dimensions
vmctl click  <vm> X Y [--button right|middle] [--double]
vmctl move   <vm> X Y
vmctl scroll <vm> [X Y] --amount N      positive = down
vmctl drag   <vm> X1 Y1 X2 Y2
vmctl key    <vm> <chord>               ctrl+alt+delete, f5, super, enter
vmctl type   <vm> "text"                US layout, ~50 chars/s
vmctl exec   <vm> [--raw] -- <command>  headless; needs the guest agent
vmctl put-file <vm> SRC DEST            host file -> guest
vmctl get-file <vm> SRC DEST            guest file -> host

Both the short key and the libvirt domain name are accepted as <vm>.

Screenshots are cached under ~/.cache/vmctl/ at mode 0700 — deliberately not under a project directory, which may be shared into the guest.

---

Using it with an agent

Claude Code: install.sh symlinks skill/SKILL.md into ~/.claude/skills/vm-use/, and it is discovered automatically. Ask it to "use vm use" or "click that button in the VM".

Any other harness: vmctl is a plain CLI, so anything with a shell can drive it. The GUI loop additionally needs the harness to render a PNG into the model's context mid-task — verified working with Codex CLI (codex exec), which needs the libvirt socket permitted in its sandbox.

skill/SKILL.md is Claude Code's file format, but its body is ordinary markdown. Paste it into AGENTS.md, a system prompt, or the equivalent, so the agent gets the loop discipline and not just the command list. That discipline — one action per screenshot, verify before proceeding, never blind-retry — is the difference between an agent that works and one that flails in a live desktop.

---

Things that will bite you

The console is not the RDP desktop

Screenshots capture the QEMU console: the virtual monitor, not a remote-desktop session.

• Windows: connecting over RDP moves the session to the RDP stack and locks the console, so your screenshots show the lock screen. You and a remote user cannot both have the desktop. vmctl doctor tells you which it is, and vmctl console-claim <vm> moves a disconnected session back — but ask first, it disconnects whoever is on it.
• Linux: xrdp opens a separate X session on :10+, so the console shows the display-manager greeter unless someone logged in locally or autologin is set.

A lock screen or greeter on the first screenshot is normal, not a fault.

Other things

• Typing is scancode-level and assumes a US layout — input-send-event does no keymap translation. For anything long or non-ASCII, use put-file.
• exec cannot launch GUI apps on Windows. The agent runs as LocalSystem in Session 0; anything graphical it starts appears on the invisible service desktop. Mapped network drives are invisible to it too, and UNC fallbacks to the same share generally fail — session 0 holds no credential for them.
• No clipboard. Session 0 isolation on Windows, Wayland on Linux. Use put-file.
• A Linux guest needs a real X input driver. A guest built for xrdp gets xrdp's own xrdpkeyb/xrdpmouse drivers, which take input from the RDP protocol rather than evdev — so no xserver-xorg-input-* package is installed and console input in X silently does nothing. The Xorg log says No input driver specified, ignoring this device, and xinput list shows only the virtual XTEST devices. Fixed with apt install xserver-xorg-input-libinput. Worth checking first on any guest where exec works but GUI input appears dead.
• vmctl flags must come before the VM name for exec: argparse's REMAINDER swallows everything after it, so exec <vm> --timeout 180 -- cmd would pass --timeout to the guest and silently use the default. vmctl errors on this rather than doing the wrong thing quietly.

Troubleshooting

Symptom	Cause
no vm-use guest registry found	No config file. The error lists every path searched; start from vms.example.toml
Unknown VM	Not in the config. vmctl list shows the valid names
NO SUCH DOMAIN in vmctl list	libvirt_name does not match virsh list --all
cannot connect to qemu:///system	Not in the libvirt/kvm groups, or not re-logged-in since being added
is shut off, not running	vmctl will not power on a guest. Start it yourself
Clicks do nothing, exec fine	Missing <input type='tablet' bus='usb'/>, or (Linux) no X input driver
guest agent ... not responding	Step 5 not done, or the virtio channel is missing from the XML
Entirely black screenshot	Booting, or the display slept. Waking it takes two steps — see skill/SKILL.md
Typed text garbled	Guest keyboard layout is not US. Use put-file
another vmctl is driving	A concurrent invocation holds the lock. Wait

---

Watching an agent work

The console framebuffer that vmctl shot captures can also be viewed live over VNC — that is how you watch without interfering. Give the domain a VNC graphics device with a fixed port and point a read-only viewer at it.

Two things matter:

• Fix the port (autoport='no'). Autoport assigns in boot order, so booting your guests in a different order silently swaps which VM each port shows.
• Make the viewer read-only. A normal viewer's clicks fight the agent for the same pointer. Guacamole's VNC connection supports read-only=true and cursor=remote, the latter letting you see where the agent's pointer actually is. QEMU itself does not enforce read-only, so anything with direct access to that port has full input — do not expose it.

Do not watch over RDP. On Windows that moves the session off the console and locks it: you would be taking the desktop away from the agent, not observing it.

---

Design notes

Non-obvious things, each verified against QEMU 8.2 / libvirt 10.0:

• Mouse press and release must be separate QMP calls. Batched into one, hid_pointer_sync sees buttons_state unchanged, sets event_compression, and merges the press away. QMP still returns {"return":{}} — a click that reports success and never happened.
• Absolute axis max is 0x7FFF. QEMU's own qapi/ui.json documents 0x7ffff; that is a typo. The conversion mirrors QEMU's VNC server (px * 32767 // width, divisor is the width) so clicks land where a human's VNC clicks would.
• HMP mouse_move is unusable here. It emits relative events unconditionally; a USB tablet only carries an ABS mask, so they are dropped. This is why the tablet in step 2 is mandatory.
• Input queues are 16 deep and drop silently — PS/2 keyboard by byte, USB-HID pointer by slot — and QMP holds the BQL so the vCPU cannot drain them mid-call. Hence one character per call with 20ms spacing.
• virsh screenshot returns PNG, not PPM, whenever QEMU advertises the capability and links libpng. Do not hardcode either; sniff.
• No raw QMP passthrough, deliberately. qemu-monitor-command reaches migrate exec:, object-add and human-monitor-command — arbitrary code execution with QEMU's privileges. Every payload is built from typed arguments.

Files

• vmctl — the CLI
• vm_registry.py — config loader; importable by other tools that want the same guest list
• vms.example.toml — annotated config template
• skill/SKILL.md — the agent-facing skill
• install.sh — the symlinks from step 3