Repository navigation
Releases
v0.6.0 — a secret the agent passes without reading
New: use-computer secret get NAME prints one stored secret, raw and with no trailing newline, for a shell substitution:
playwright-cli fill e12 "$(use-computer secret get gh-token)"
type --secret goes through the keyboard, so it needs focus — the wrong channel for a browser driven by Playwright, which fills a field by selector and only needs the value. The shell expands the substitution; the agent sees the command, never the value. Resolution and errors are the same as --secret.
The bundled skill teaches that secret get runs only inside $(...), never bare, and that the receiving command's output (echoed arguments, generated code, page snapshots) is context too. Run use-computer skill update to pick it up. type --secret and set-value --secret are unchanged. See CR-024.
v0.5.2 — no real name in the example output
The example `windows` output carried a real Telegram contact's name instead of a made-up one -- captured live while measuring an early read (see CR-007) and never swapped for a placeholder. It has been in the bundled skill, and so in every published wheel, since 0.2.0. Replaced with a generic name everywhere it appeared (the skill and three docs). No behaviour changes.
v0.5.1 — zoom when a guess is not enough
`screenshot` can now crop and magnify around a guessed point, not just a tree node:
```bash
use-computer screenshot --x 860 --y 420 # 200x200 around a guess (--radius 100)
use-computer screenshot --x 860 --y 420 --zoom 4 # and magnified
```
This is the case `--of` cannot reach: a canvas, a map, a game, a VNC session -- no tree to crop to. A screenshot there is usually downscaled before a vision model sees it, so a coordinate it hands back is a guess at the shrunk scale, and turning it into a real pixel means multiplying by a resize factor nothing prints anywhere -- exactly the arithmetic a click goes wrong on. The result carries the crop's pre-zoom box and the zoom factor, so converting a point read off the zoomed picture back to a screenshot coordinate is one division and one addition instead.
It is a tool for the guesses that need it, not a mandatory stage of the ladder: a small or crowded target, a click that already missed, a guess not worth trusting as it stands. Most rung-three clicks still go straight from a guess to a click, and the skill teaches it that way.
Also in this release
A mistyped command such as `use-computer bad-command --help` was silently rewritten into `batch bad-command --help` and showed batch's help -- or, without `--help`, failed with a file-not-found for a file nobody meant -- with nothing anywhere saying `bad-command` was never a recognised command. The default-command rewrite now only fires for an actual batch source, `-` or a path that exists; anything else reaches typer's own dispatch, which says plainly `No such command 'bad-command'.` (BUG-021)
v0.4.0 — a secret the agent never reads
A password or a token is stored once, by name, and typed by name — so it never enters the agent's context, its transcript, or the shell history.
use-computer secret set gh-token # the value comes from stdin, never from argv
use-computer type --secret gh-token
use-computer set-value --id 0/2/1 --secret gh-tokenOne invariant holds it together: a secret leaves the store into the keyboard, and never into stdout.
- There is no
secret get, and its absence is the design — a command that printed a secret is one an agent would call. - The result line trades the text for the name:
type 40 chars (secret gh-token). - An action carrying a secret is never verified, because the after-screenshot would be a photograph of the credential. The line says
not verified (secret)rather than going quiet. - A missing secret names the command that stores it and tells the agent not to ask for the value in the conversation.
- In a batch,
secretreplacestextorvalue, so the file names the credential without containing it.
The bundled skill teaches the mechanism rather than the flag: an agent shown only --secret will helpfully offer to store the password it was just handed.
Encrypted at rest
Values are Fernet tokens. The key lives in the XDG data directory while the ciphertext sits in the config one — a key beside the data is theatre, since whatever copies one copies the other. USE_COMPUTER_SECRET_KEY_FILE moves it. Names and timestamps stay readable so secret list needs no key.
This stops the way credentials actually leak — a ~/.config that gets synced, committed to a dotfiles repository, or pasted into a bug report. It does not stop a process running as you, which can read the key or simply run type --secret. Both halves are in the documentation, because "encrypted" is a word a reader completes with their own threat model.
cryptography is a core dependency rather than an extra: a store that is sometimes encrypted is not a property anyone can plan around.
Also in this release
The Windows work tagged as 0.3.5 and never published: the window in front is recognised there, a window id no longer drifts as activate reshuffles the Z-order, a batch piped in from a Windows shell survives its byte order mark, and an elevated window says that is why it refuses.
v0.3.4
Everything since v0.3.2: 0.3.3 was bumped in the tree but never tagged, so its contents ship here.
The skill an agent can actually follow (CR-020)
The bundled skill no longer mentions --use. A profile names which machine and which backend, and no result an agent can read distinguishes one from another — so five of thirty-one examples carrying --use laptop taught a model to invent a profile name for whatever machine it was running on.
- A single declared profile is selected without being named. Counted across the project and global config files.
config showattributes it to the layer that declared it.allow-localremains a separate opt-in, so a lonelocalprofile that has not given it is selected and still refuses. - The refusal names one next move per state instead of a three-alternative menu: no configuration, a config declaring no profiles, or several profiles with no default.
- The skill leads with a six-step procedure —
windows,activate,tree, act by name,treeagain, pixels only if the tree cannot see it — executable without reading further. The reasoning follows it rather than preceding it, which matters for a smaller local model that acts on what it reads first.
Fixes
- BUG-020 — a macOS node id no longer drifts to a different element between two reads.
- BUG-018 —
activatemakes the application frontmost on macOS instead of reporting success without coming forward. - BUG-019 —
--windowalone is not a selector, and no longer crashes when paired with a coordinate. - BUG-017 — a click's press waits for macOS to catch up with the move.
- BUG-015 — the desktop is read once per
windowslisting.
v0.3.2 — a depth limit is not a blind spot
Found while verifying 0.3.1 from PyPI: at a shallow --depth the ?unexposed marker appeared on a node whose children the depth limit had cut. At the depth a snapshot stops, every node looks childless, so the marker there reports the caller's own limit as a property of the application — the same mistake BUG-011 fixed for 'empty', in a feature shipped an hour earlier. The boundary level is no longer marked.
v0.3.1 — ask the window manager
AT-SPI cannot say which window is on top: it reports 'active' per application, so several windows claim it at once. 0.3.0 answered that honestly by marking none — which made 'use-computer tree' with no arguments stop on an ordinary desktop to ask something the operating system already knows.
It is not AT-SPI that knows it. On X11 the window manager publishes _NET_ACTIVE_WINDOW, and it is unambiguous. A provider now offers that as a hint and the matching happens above it, so all three platforms share one rule. The refusal does not go away; it stops being the common case.
python-xlib is imported softly and joins the Linux 'tree' extra, which had been empty since BUG-003 for a reason that does not apply to a package with no compiled parts. Absent, or on Wayland, everything behaves as before.
And the 'skill' commands answer in text. 0.3.0 said 'every command, without exception', named two, and missed the one an agent runs to find out whether its own instructions are current.
v0.3.0 — a result you can trust
Bugs found by running the tool rather than reading it: --verify denied changes you could see (256px compare, 0.2%-of-the-frame threshold); every window was 'active'; a --depth you asked for came back as 'empty'; screenshot --of cropped whatever was in front; an element action reported a success the platform had ignored.
New: ?unexposed names the region an accessibility tree does not describe -- a canvas is in no tree at all, and nothing marked the hole. 'activate' and --window on the coordinate actions, so a coordinate stops landing wherever the pointer happens to be. Every result line now carries what it did: the key combination, the typed text and its length, both ends of a drag, the coordinate actually sent. --version and config show are text like everything else.
v0.2.2 — verify speaks, screenshots live with the project
Fixed
-
--verifyreported nothing. Inverting the output contract in 0.2.1 dropped both things the
flag exists for. It says them again:click at (200, 200) — changed 6% — 41 ms 20260905T093017.762Z-click.pngunchangedis printed as a word, not afalse: it is the one an agent has to notice, because it
means the coordinate was stale. -
Every run now ends with what the envelope carried, for thirteen tokens against its 177:
ok — profile laptop, backend local, screen 1920x1080, scale 1 failed at action 3 — profile laptop, backend local, screen 1920x1080, scale 1Cheap must not mean lossy.
scale unknownprints as words, because refusing to guess is only
useful if the caller can see it happened. stdout says what happened, stderr says why. -
A repeated screenshot directory cost more than the whole summary line. A path is 23 tokens
and a batch of five verified actions repeated the directory in all of them. It is named once now,
and the lines carry filenames.
New
-
Screenshots live with the project, in
.use-computer/screens/, with the XDG data directory
as the fallback when there is no project. The tool writes.use-computer/.gitignorewhen it
creates the directory — a capture is the whole desktop, and a working tree is onegit add -A
away from committing it. That also closes a gap older than this release:.envhas been
described as gitignored since the beginning and nothing ever made it so. -
use-computer pruneremoves them, with--dry-runand--keep N. It deletes only files
matching the names it writes, never the directory, and counts what it left alone.
The bundled skill
Rewritten around the two ways of knowing — structure and pixels — what each reaches, what each is
blind to, and which error means go and look. --via, the flag that chooses which rung of the
ladder an action takes, is finally in there. A test now asserts that everything the skill mentions
actually exists.
v0.2.1 — text is the output
Breaking: stdout is text, not JSON
Measured on this tool, in tokens: screenshot was 243 as JSON and is 4 as text; a click
on an element 4,238 against 21; the envelope alone, contents removed, 177 before
anything is said — spent on profile and backend (constant for a session), screen (five
numbers nobody reads), ok and failed_index (which the exit code already carries), and the
action echoed back to the caller who just sent it.
The CLI's consumer is a model, and a Python API exists underneath for programs. It had been
optimised for a parser that has somewhere better to be.
--format json returns the previous envelope, one flag away, and is never inferred from
isatty(). Exit codes and the Python API are unchanged.
New
screenshot --of <node-id>crops to one element: 18,284 pixels instead of 2,073,600 for a
question about a 653×28 field. The cheap way down to vision when an application paints its own
text rather than exposing it.windowsreports the application, so two windows both titled "ChatGPT" can be told apart.
It refuses rather than guesses
--windowno longer picks the first title match. It lists the candidates. This is not a rare
case: a terminal puts the running command in its own title, so the terminal executing
--window "X"matches X — and the first match was usually the window you were looking at rather
than the one you named. Pass the window id fromwindowsto be exact.
Fixed
- One unresponsive application no longer takes down a whole desktop read.
- A missing required option exits
2, not1with a traceback, on the supported typer floor. richno longer eats[tree]out of the error whose whole job is to name it.- The
treeextra no longer promises Linux something pip cannot deliver; the error now inspects
the installed bindings and says whether they are absent, invisible, or built for another Python. - Window matching had drifted into the Linux provider. It is policy, it now lives above the
Protocol, and all three platforms share it.