Repository navigation
Releases: mcp-tool-shop-org/ai-playtest
Releases · mcp-tool-shop-org/ai-playtest
Release list
v0.1.0 — family-diverse AI playtesting
First public GitHub release. Dogfood swarm #2, 2026-09-14. 84 -> 182 tests,
then the Law 8 execute set. First live run against claude-rpg (proof-01,
8 turns, mistral). Stage B: 24 HIGH. Stage C: remaining MED/LOW
(humanization — errors that name the cause, CLI help, report honesty).
Feature execute: consume Observation.actions, serial RPC reset(),
viewport-only PTY grid, report legends, schemaVersion on leaving
artefacts, single-run REPORT.json.
Not on npm (package.json stays "private": true).
Added
- Typed actions from
Observation.actions. Player replies map onto
choose/key/line. A closed-set miss is a harness event
(illegal-action); the runner does not step.describeActions()is appended
to the player prompt. - Serial RPC reuses one driver.
--serialondriver.kind === "rpc"
starts once, callsreset()between seats, and stops once. A missing or
throwingresetfails that next seat withE_RESET(no silent second
client). Parallel RPC stays one process per seat. - PTY viewport grid.
grid.linesisviewportY .. viewportY+rows-1, not
the scrollback buffer. Prompt matching usesbaseY+cursorY.kind:key
writes without a trailing CR. - Single-run
REPORT.jsonsidecar (kind: "single-run-report") next to
REPORT.md. Markdown legends teach!,repeat,loop, andH(a)in
bits.schemaVersion+reportFormat+VERSIONare stamped on the
markdown generator line, both JSON reports, and seatmeta.json. - State-gated invariants. HP never negative and inventory non-decreasing,
only when a turn carries a structuredstateobject — never inferred from
prose. - Landing page + Starlight handbook (
site/, accent cyan). Pages workflow
path-filtered tosite/**. Live at the org GitHub Pages URL. - README translations (ja, zh, es, fr, hi, it, pt-BR) via TranslateGemma 27B.
- README lockup from
mcp-tool-shop-org/brand(logos/ai-playtest/readme.png). - Deterministic transcript verifiers (
src/verifiers.ts). Six checks, all
transcript-only: absorbing-SCC (Tarjan over the observed screen digraph,
kind=review, never a trap proof), ignored-input attribution, parser
unparsed/refused/accepted with empty defaults, terminal victory/death,
no-progress windows, entity-appearance leads for the jury. Specified in
docs/research-2.md§D; false-positive modes printed beside every hit. --runs Nwith a Beta-Binomial posteriorBeta(s+1, n−s+1), stability
labelsSTABLE_PASS/STABLE_FAIL/UNSTABLE, worst-of-n beside the mean,
and the copy that n=3 is descriptive (2/2^n= 0.25). First n that can
clear α=0.05 is 6. Do not bootstrap.- Jury
n_effon the report:k / (1+(k−1)·φ̄), warn whenn_eff/k < 0.5. - RPC
hello {protocol:1}is a hard handshake. A bridge that only
speaksobserve/act/quitnow fails closed on connect. Serial
runAllnow callsreset()between RPC seats. - stdio game driver (
stdio-game.ts) — spawns a game as a child process and
decides when the game is waiting for input: apromptPatternsregex matched
the stripped tail and output went quiet forpromptQuietMs, or nothing matched
andidleQuietMsof silence passed. The runner never parses the game. - OpenRouter seats (
openrouter.ts) — one HTTP seam, injectablefetch, six
retries with exponential backoff (126 s total) for bursty upstream rate limits. - Model players (
player.ts) — a seat sees the screen since its last input
plus its lastplayerMemoryTurnsexchanges, is briefed bypersona(goals and
register, never the mechanics under test), and answers with one line. - Scripted setup (
config.ts,run.ts) — prompts matching asetup[].match
regex are answered from the config without spending a turn, so every family
starts from the same character. - Critic (
critic.ts) — reviews a transcript against the game'scriteria[]
at temperature 0 and returns one JSON object: per-criterion met/evidence/turn,
an alive verdict, highlights, dead spots, confusions. - Report (
report.ts) — aggregates every seat: criteria by family, verdict
counts, and every dead spot and confusion named with the seat that found it. - CLI (
cli.ts) —runandreportverbs; exit codes 0 ok, 1 usage,
2 config, 3 provider, 4 run error. - Config validation refusing two seats from one model family.
Changed
- The runner consumes
actions/state. Illegal-action rejection and
hp/room hashing are wired;docs/engine-bridge.mdno longer says they are
protocol-only. panelSizedefault 1, not 3. Kohli 2026 (Kish n_eff 2.18, panel 72.0% vs
best single 71.8%, cross-family φ 0.389 vs same-family 0.437) undercuts
family diversity buys independence. Author-off-jury stays (Panickssery /
Stechly). Extra jurors flag disagreement. Decision written in
docs/research-3.md.- Coverage "thin" on a short-but-varied session is labelled a sample-size
limit, not "explored thinly / never reached the content" (proof-01). - Criteria-table agreement uses juror splits on a 1-seat run (proof-01 hid
1/3 splits behindno!and an em-dash). - Jury dead spots and confusions appear in the report, not only the author's.
- Three drivers behind one observation seam.
stdio(the original),
pty(a rendered terminal grid, for full-screen TUIs), andrpc(structured
state over TCP, for Godot / Unreal / anything instrumented). Selected with
driverin the config; defaults tostdio, so existing configs are
unaffected.docs/engine-bridge.mdcarries a paste-and-go Godot 4 autoload. - A cross-family jury. Each transcript is judged by author-off seats
(panelSize, default 1). Majority verdict, split criteria marked, dispersion
reported. (Shipped at default 3 in swarm #1; default moved to 1 in swarm #2 —
see Changed.) - Coverage. Novelty curve and half-life, repeat / loop / self-loop rates,
action entropy and a thin/moderate/broad read, computed from turn records
alone. LICENSE(MIT),SECURITY.mdwith the threat model, CI,prepublishOnly,
tsconfig.test.json, and this changelog.
Fixed
- Last player input's resulting screen was labelled
quit. The quit loop
consumed it, so the critic never saw the last action's result (proof-01:
/directoropened director mode; the recap the jury cited was the save
screen). Consequence is now recorded first; runner quit inputs stayquit. - Verdicts could be inverted.
parseCritiquecoerced withBoolean(), and
Boolean("false") === true, so a critic answering"alive": "false"as a
string was recorded as alive with every criterion met. - The OSC branch of the ANSI regex over-deleted. It terminated on the next
BEL anywhere in the buffer, so an ST-terminated OSC plus a later BEL removed
everything between them — 46 characters of narration in the measured case.
That regex also stored literal control bytes, so reviewers saw a different
pattern than the one that ran. OPENROUTER_API_KEYreached the game process and was observed rendered
into screen text, from there into the player's context and the written report.
The child now gets an allowlist;game.inheritEnvopts back in.- The critic's evidence included the runner's own quit inputs as player
decisions and excluded every terminal screen, so crashes and stalls were
invisible to the verdict. - The transcript sat last and unframed in the critic's prompt — the most
instruction-weighted position — so a game could steer its own grade. - stderr shared a quiet-timer with stdout (3063 ms timeout every turn where
220 ms prompt was correct); stdout chunks were decoded independently,
corrupting split multi-byte characters; an unhandled EPIPE on stdin took down
every parallel seat; a failed spawn reported nothing. finish_reason: 'length'was returned as success, andres.text()sat
outside the retry loop's try/catch.--label ../../etcescapedrunsDir;--turns abcplayed zero turns and
exited 0;Promise.alldiscarded every sibling when one seat threw; a run
where every critique failed exited 0; the report read "Alive verdicts: 1 of 1"
for a two-seat run with one dead seat.- Critic token budget raised 1,800 → 6,000. Nineteen criteria with evidence
strings no longer fit in 1,800 tokens; one seat's critique was cut mid-array
and failed to parse twice. Sized for roughly 25 criteria. Retries now carry the
previous parse error so the second attempt is informed.