Skip to content

v0.1.0

Choose a tag to compare

@crftwr crftwr released this 17 Oct 17:03
· 36 commits to main since this release

quality-loop 0.1.0

The first release. quality-loop measures what an agent-facing product teaches:
a skill bundle, an MCP server or a CLI, used by an agent that has nothing else.
Each task runs in a room that holds the product's agent-facing surface and
nothing more, started so it cannot read the repository the answers are in. A
second agent, the operator, puts up the screen the task needs. Afterwards it
reports the problems it found, each with a severity. There is no pass/fail.
The report is for a human to act on.

  • Scenarios are Markdown. TOML front matter, then ## Prompt, ## Screen,
    ## Must and ## Must not. The prompt is copied into the room verbatim by
    code, never by an agent. A scenario whose prompt carries one of its own Must
    lines is refused, because that is the scoring key leaking.
  • Rooms are isolated by flags and checked by an audit. --restricted, an
    explicit permission list under dontAsk, exactly the product's MCP servers,
    git stopped at the room, and stdin closed. Afterwards the transcript is
    audited for reaches into the checkout, the installed package, the user's
    set-aside files, or another author's work. A refused reach is reported. Only
    one that came back with something is a breach.
  • The user's files are stood in for and always put back. Fixtures replace
    the product's files for each room. Everything the room made is collected as
    evidence, and the originals return in a finally that Ctrl-C and SIGTERM
    unwind through. A stand-in the room could reach refuses the room.
  • The operator has no shell. Its tools are Read, Glob, Grep, writes to its
    own directory and the room's REPORT.md, and quality-loop's own MCP server.
    That server opens and closes screens the product declares by name, and
    runs the product's declared evidence commands with no shell. It never uses
    the product's MCP, so a broken product cannot hide a failed setup.
  • Screens on macOS and Windows. Pages are served over a local HTTP origin
    and record what is done on them. Terminal scripts are Python, run in a fixed
    shell without the user's startup files. On macOS this uses Safari and
    Terminal. On Windows it uses Edge and Windows Terminal; the Windows backends
    have not yet been run on a Windows machine.
  • Reports feed a loop. Each problem has a stable id, a kind (skill gap,
    product, harness, out of scope), its evidence, where it would be fixed, and
    how a fix is confirmed. FINDINGS.md and findings.json gather every room,
    most severe first.

examples/toy/ runs the whole thing against a product small enough to read
in a minute. Keyhac is the first real product (crftwr/keyhac#205).