v0.5.0

quality-loop 0.5.0

A product's test suite runs in its guests. quality_loop.guest.suite.run_pytest runs pytest in a guest the product has staged - in a Windows VM's desktop session, over SSH on macOS, through docker exec in a container - prints its output as it comes, and keeps the log and JUnit XML. Keyhac and XeFM each carried a copy of this; both now use it.

Linux guests are Docker containers. A docker = "<image>" table in guests.toml makes the guest a container for the run: the share is a bind mount, commands go through docker exec, and the container is removed afterwards. Guest.system tells Darwin, Linux and Windows apart - a non-Mac guest used to be taken for Windows.

Two pytest marks for where a test may run. -p quality_loop.pytest_plugin gives desktop (skipped unless --desktop, for tests that take focus, move windows, send input or change the clipboard or trash - a VM's run passes it) and host_only (skipped in a guest; --host-only runs just those).

Also: a terminal app driven in a pty is no longer read before it has drawn anything, so a slow start is waited for instead of recorded as an empty screen.

For an adapter: nothing has to change. A product that defined --desktop in its own conftest drops it when it loads the plugin, or pytest refuses the duplicate option.

Full Changelog: v0.4.2...v0.5.0

quality-loop 0.4.2

quality-loop 0.4.2

A VMware guest boots for every run. Take the guest's snapshot with the VM shut down: each run reverts to it and boots, as each Tart clone does. A snapshot taken while the VM ran comes back in seconds, but with whatever was half done at that moment, every time. On Windows the run now waits until the automatically signed-in desktop has been up for 15 seconds before it starts anything there. SSH answers before the sign-in has made that session. If no desktop comes, the error says to set up automatic sign-in. A running-state snapshot passes the wait at once, so an existing image keeps working unchanged.

vmrun takes the snapshot. A vTPM-encrypted VM cannot be snapshotted by vmrun while it runs, but it can once it is shut down: vmrun -vp <password> snapshot <vmx> <name>. docs/guests.md says what the image needs: automatic sign-in, and nothing covering the desktop at sign-in.

Nothing changes for an adapter.

quality-loop 0.4.1

quality-loop 0.4.1

A run keeps the machine awake — while a run goes, quality-loop holds off this machine's idle sleep (caffeinate -i on macOS, SetThreadExecutionState on Windows) and says so in the log with an awake: line. A Mac left alone on battery used to sleep through a run — one spent 40 of its 43 minutes asleep — and a VM guest stopped with it. The display may still sleep, and closing a laptop's lid still sleeps the machine.

Full Changelog: v0.4.0...v0.4.1

quality-loop 0.4.0

quality-loop 0.4.0

Run the product's app in a VM — the new quality_loop.guest layer starts a Tart clone or reverts a VMware snapshot, stages the product under test on a shared folder, builds a per-run venv in the guest from a pip cache on the share, starts the app in the guest's desktop session, and tears it all down afterwards. One settings file, ~/.config/quality-loop/guests.toml, describes this machine's VMs for every product. The app runs off your keyboard and screen, and your own copy can keep running. See docs/guests.md.

Screens go up in the guest — an adapter whose app runs in a VM returns quality_loop.guest.screens from screen_backend(), and the operator's pages and terminals open on the guest's desktop (Safari / Terminal on macOS, Edge / Windows Terminal on Windows). The pages are still served from the host, through a reverse tunnel.

Also: Keyhac is the first product on the layer (Windows 11 on VMware Fusion and macOS on Tart, both run end to end). XeFM follows.

Full Changelog: v0.3.0...v0.4.0

quality-loop 0.3.0

quality-loop 0.3.0

Screens can go up where the product runs. An adapter whose product runs somewhere other than this machine (in a VM, for example) can now answer Adapter.screen_backend() with a module that opens and closes the operator's pages and terminals there, so the room's app sees them on its own desktop. The runner records the backend in the screens manifest, and screen_open / screen_close and the end-of-run cleanup all go through it. The default (None) keeps this machine's backend. docs/adapters.md describes the module's three functions.

The scenario's platform comes from the adapter. Adapter.platform() names the OS the product runs on (default sys.platform). Phase-one operator instructions use it, so a Mac host driving a Windows guest gets Windows instructions.

No adapter has to change: both methods are new, and their defaults keep 0.2.2's behaviour.

v0.2.2

quality-loop 0.2.2

A note after a finding's id is no longer part of it. Operators given a known list sometimes write - **Id:** dialog-gone (known). The " (known)" used to become part of the id, so a problem seen again was counted as new: the first Keyhac run with a known list reported 34 new findings and 23 still there, when its reports held 17 new and 40 seen again. The id is now the first code span on the line, or else its first word; docs/reports.md says so.

No adapter has to change.

quality-loop 0.2.1

quality-loop 0.2.1

A lone Esc is no longer glued to the next key. A terminal app waits a moment after an ESC to see whether an escape sequence follows: PuiKit waits 0.1s, and ncurses waits ESCDELAY, a second by default. So an Esc followed quickly by another key reached the app as Alt plus that key. The first live check of XeFM sent Esc and then Ctrl-T 20ms apart, and the command palette never opened. The pty backend now pauses after a lone Esc, and starts the app with ESCDELAY=25 unless the adapter sets its own.

Report fields may continue as a nested list. An operator that wrote - **Evidence:** and then put the evidence in an indented list below it had that finding listed as malformed, with "no evidence". Indented lines now continue the field above them.

Tests pass on Windows. quality-loop has now run end to end on Windows (11, ARM64) against the toy, Keyhac and XeFM, and the screen backends have opened and closed real pages and terminals there. Driving a desktop app with synthetic keys (drive.DesktopApp) has not yet run for real on either OS, and a terminal app is not driven on Windows yet.

No adapter has to change.

quality-loop 0.2.0

quality-loop 0.2.0

A command line for running and following the loop. quality-loop run starts a run in the background and prints its id. status, watch, findings, cancel, restore and check take it from there. findings compares a run with the one before it, by room and id: new, still there, gone, back. qa/quality-loop.toml names the product's adapter.

Runs you can watch from another terminal. Each run writes run.log and status.json beside its rooms: every room's phase, its cost, when it was last active, and whether it is quiet but alive. A room that reaches into those files is an isolation breach.

Rooms side by side. An adapter can declare parallel, resources and needs() to run rooms at once without sharing what they must not. An adapter that declares none of them runs one room at a time, as before. A room that fails gets a report and the others go on.

The framework drives the product's app. drive.TerminalApp runs a terminal app in a pty and reads its screen as text (pip install 'quality-loop[tui]'). drive.DesktopApp drives a GUI with synthetic keys, checking before every key that the app's window is in front, and reads it by screenshot. A scenario's new Live check section has the operator drive the real app after the room is done. With room_apps, a human-facing product's room drives the app with only its manual. Every step is recorded with its intent and the screen afterwards. An app that writes JSON lines to QUALITY_LOOP_EVENTS gets level-1 evidence. examples/tui/ measures a terminal app and its manual with a few dozen lines of adapter.

A conductor skill. quality-loop init installs a skill into the product's .claude/skills/, stamped with this version. With it, a Claude Code session in the product's repository can set quality-loop up, plan runs with a person, file findings as Issues, fix what the person chooses and retest. init --upgrade refuses to overwrite a copy that was edited by hand.

Known problems as neutral data. run --known gives each operator the ids and titles of problems earlier runs reported for its scenario, never a status. Reports gain an optional Reproduce field, and each room's evidence/run.json records where it started: product version and commit, OS, fixtures and models.

For adapters

Nothing has to change: every addition is optional. A few new hooks are worth adopting:

• default_variant names the unnamed variant for --variant.
• restore() backs quality-loop restore.
• bundle_place = "docs" puts a manual where a skill would go.

QUESTIONS.md now starts "What I was not told". findings.json entries gain reproduce and run.

Windows: the key and screenshot backends are unit-tested only, and terminal apps are not driven there yet.

v0.1.0

quality-loop 0.1.0

The first release. quality-loop measures what an agent-facing product teaches:
a skill bundle, an MCP server or a CLI, used by an agent that has nothing else.
Each task runs in a room that holds the product's agent-facing surface and
nothing more, started so it cannot read the repository the answers are in. A
second agent, the operator, puts up the screen the task needs. Afterwards it
reports the problems it found, each with a severity. There is no pass/fail.
The report is for a human to act on.

• Scenarios are Markdown. TOML front matter, then ## Prompt, ## Screen,
  ## Must and ## Must not. The prompt is copied into the room verbatim by
  code, never by an agent. A scenario whose prompt carries one of its own Must
  lines is refused, because that is the scoring key leaking.
• Rooms are isolated by flags and checked by an audit. --restricted, an
  explicit permission list under dontAsk, exactly the product's MCP servers,
  git stopped at the room, and stdin closed. Afterwards the transcript is
  audited for reaches into the checkout, the installed package, the user's
  set-aside files, or another author's work. A refused reach is reported. Only
  one that came back with something is a breach.
• The user's files are stood in for and always put back. Fixtures replace
  the product's files for each room. Everything the room made is collected as
  evidence, and the originals return in a finally that Ctrl-C and SIGTERM
  unwind through. A stand-in the room could reach refuses the room.
• The operator has no shell. Its tools are Read, Glob, Grep, writes to its
  own directory and the room's REPORT.md, and quality-loop's own MCP server.
  That server opens and closes screens the product declares by name, and
  runs the product's declared evidence commands with no shell. It never uses
  the product's MCP, so a broken product cannot hide a failed setup.
• Screens on macOS and Windows. Pages are served over a local HTTP origin
  and record what is done on them. Terminal scripts are Python, run in a fixed
  shell without the user's startup files. On macOS this uses Safari and
  Terminal. On Windows it uses Edge and Windows Terminal; the Windows backends
  have not yet been run on a Windows machine.
• Reports feed a loop. Each problem has a stable id, a kind (skill gap,
  product, harness, out of scope), its evidence, where it would be fixed, and
  how a fix is confirmed. FINDINGS.md and findings.json gather every room,
  most severe first.

examples/toy/ runs the whole thing against a product small enough to read
in a minute. Keyhac is the first real product (crftwr/keyhac#205).