Skip to content

Releases: awss1i/assay

Release list

v0.1.3

Choose a tag to compare

@awss1i awss1i released this 21 Sep 17:43

Install docs. The README now says Python 3.10 or newer next to the install
command, points at python3 -m pip when pip is not on PATH, and offers
uv tool install assay-ui / pipx install assay-ui for an isolated install.
No code changes.

v0.1.2

Choose a tag to compare

@awss1i awss1i released this 21 Sep 13:33

assay opens a generated web page in a real browser, works out a test plan from
what the page renders, drives every control it finds, and reports what it
pressed and what happened.

No tests to write, no baseline images, no model in the loop. It gives the same
answer twice.

pip install assay-ui
assay ./my-app

What it finds

Two benchmarks, built differently, both checked into the repository.

225 generated programs, written to 75 objectives by three harnesses. A
person opened every one and drove it before assay saw it; 20 are broken.
assay found 15 of the 20, and raised 0 false alarms across the 205
that work.

10 working programs and a copy of each with 5 bugs planted by a different
harness and model. The harder set: pages that work and are wrong, rather than
pages that stopped. assay found 10 of the 50, and flagged 0 of the 10
working originals
.

Both reproduce with python bench/score.py and python bench/planted/score.py.
No key, no network.

What it says

It never reports that a page is broken. It reports what it pressed and what
happened:

C015 [FAILED] draw on canvas twice, then press Undo twice
    → the first press did nothing and the second did something, from the same
      state, so this control is one behind

In a harness

Six harnesses install assay with their own command, and the skill is one
markdown file for the rest of the fifteen. Claude Code and DeepSeek
Harness additionally take a plugin, which runs the check at the end of any turn
that touched a page, whether or not the agent thought to.

Limits

It checks browser programs, one page at a time, built rather than source. It
cannot judge intent: a control that works mechanically and does the wrong thing
passes. The README says where the edges are.

MIT.