Repository navigation
Releases: ArtJack/verdict
Release list
v0.93.1 — what a team commits is the record
-
The local tier no longer writes its gateway's address into the record.
verdict-localput the
gateway's host inlast_run.local, finalize copied it intostate.json, and the report's Judge
line printed it. In team mode both are committed with the code, so every local run committed the
address, and the whole base URL when it parsed to no hostname. Now the host goes only to the bill
(usage.jsonl), as a bare hostname, never the URL around it. A facts.json written by 0.90.10 or
earlier has its host taken out, and billed, before the state is composed. The Judge line prints no
host, even from a state written before. A team record written by an earlier local run still
carrieslast_run.local.host; delete that key by hand. The signed chain row does not carry it.
This repository's own run-16 state and report no longer name it. -
Team mode keeps the bill and one run's scratch out of git. The
.qa/.gitignorethat
/verdict:baseline teamwrites now listsusage.jsonlandfindings.prev/beside the per-run
intermediates.usage.jsonlis each machine's spend and, for the local tier, the gateway it
called; nothing reads it back. A.qa/made by an earlier version needs both lines added by hand. -
The Stop hook keeps its own record of what it said.
stop_hook_activesays a Stop hook
blocked, not which one, so another plugin's continuation silenced every check. The hook now
records what it said per session and QA root, says a hand-written state once per session, and
keeps the old rule wherever no record can be kept. -
Six new hygiene leads, tier 3:
sql_from_string: a value spliced into SQL. DDL identifiers, bound parameters, constants,
casts and tagged templates are not leads.jwt_unverifiedai_key_in_browserpaid_api_from_browsercompose_db_port_publishedcompose_default_password
Found by comparing this scan with another author's MIT plugins (haraldalder-vibemogger's
vibe-audit). Each was re-measured here; none of their code was copied. -
Pinned mutants: 26 new (22 for the Stop hook and the leads, "gateway host 1–4"); badge 527/527.
v0.93.0 — the cheap run is a normal run, capped
verdict-run --lite: the cheap asked run. A normal asked run (the tester,verdict-factsbefore
it andverdict-finalizeafter, the in-run replay, questions on the ledger) under a $2.50 cap unless
--max-budget-usdnames another, with the tester told its budget: about 35 tool calls (--calls N
names another) and the dollar stop, with room kept to finalize, since a session stopped at its cap
writes no state. No breadth pass, no packet. Measured on four of the recall benchmark's repositories
(15 known defects) against plain Claude Code given the same scope, model, effort and cap and no
Verdict: plain located 11 and 10 in two runs and proved 10 and 9 for about $6.40;--litelocated 10
and proved 7 for $6.45;--escalatelocated 8 and proved 6 for $6.16. The packet's "work the list in
order" is what cost recall.--escalatestays, for comparison;--liteand--escalateare refused
together. One or two runs per arm: a direction, not a rate.- Every open finding with a repro is replayed, Major+ first: before the tester (the previous
record's findings), at finalize (this run's), and byverdict-replayby default. The proof gap above
was mostly findings filed at Minor with a repro no replay ran. A Minor's failedprovenclaim is
stamped on it and named, and the run stands; only a Major+ claim refuses a run, so a capped run is
never refused near its cap over a Minor. A cap or a time budget on the replay cuts Minors first. - Any capped asked run is told its cap and closes with its bill, not only an escalation run.
--calls Nnames the call aim on any asked run;--escalate-callsstill works. eval/recall/plain_arm.py, the standing control arm: the same case, scope, model, effort and cap
without Verdict. It runs in a standalone clone outside the home directory: a linked worktree reads its
checkout's.claude/settings.local.json, and in the first measured round an earlier Verdict run's
hooks there refused every Bash and Write. No user settings, MCP servers or web tools. The session's
findings are scored byscore_recall.py, its repros replayed by Verdict's own replay, and a scan of
its transcript flags any read of Verdict's records or harness.- Pinned mutants: 8 new (lite 1–5, replay 0.93.0 1–3), escalate 12 re-anchored; badge 501/501.
v0.92.0 — breadth for a cent, judgment from a list
verdict-run --escalate: a cheaper asked run. It measures first, as every asked run does,
then runsverdict-localin a scratch copy of the QA root on those facts, at zero Claude tokens,
and launches the tester with a packet instead of an open brief: claims the local tier proved by
counterfactual, every census survivor function, the head of the risk-ranked reading map, the small
model's unproven readings and leads, and the reasons a model run is owed. The tester proves or drops
each item and covers the scope its instructions name where the list does not reach. A $2.50 cap by
default, named in the packet; the run closes with its bill. Nothing starts, and never a full run in
its place, when the gateway is down, the measurement failed or the breadth pass did not finish.
The packet is kept asescalation.md;last_run.escalationcarries its summary into the state, the
bill and a not-tested line in the harness's voice. Measured on the recall benchmark (four
stranger repositories, 13 known defects, one baseline each): full asked runs located 9 and
proved 7 for $15.55; escalation covering the named scope located 7 and proved 6 for $6.37. With
the risk focus below, on four repositories with 15 known defects: 8 located for $6.16 against
the full runs' 11 for $15.09, plus one Major outside the known list that a hand replay
confirmed. Every packet pointed at every known defect.--escalate-calls Nnames the
tool calls the packet asks for (default 35); at 60 the tester found 7 of the 15 for $7.89, its
extra calls going to findings outside the known list, so the default stays. The benchmark's key
was rewritten with mechanism terms (its domain words credited findings to the wrong defect, one
higher per row); these numbers use the new key. One run per case: a direction, not a rate.- The scope a profile names steers the breadth pass, the reading map and the packet.
verdict-factsreads the profile's risk-focus sections into facts.json asrisk_focus: what they
name in backticks (a bareutils.pyis read in the directory its bullet named), their prose
words, and the files those point at, ranked: files named outright, then files defining the named
functions (a name defined once before one defined in a dozen;set_inputalso finds
set_input_dispatch_by_period), then files under a named path, then files whose path carries a
named word; never tests or migrations. The breadth pass reads those after the census survivors'
files, named functions first; with no coverage map it used to read the largest files. The reading
map counts them as risk paths when the profile sets norisk_paths(it used to rank the least
covered modules alphabetically). The packet lists the named functions (two per file, going round
the files, skipping stubs and bodies that only raiseNotImplementedError) and the files named
outright, and shows a module in the reading map as a module, with its coverage.
eval/recall/packet_recall.pymeasures the packet at zero Claude tokens, counting a known defect
as pointed at when the packet cites its file: on six stranger repositories with 17 known defects,
it points at 16, up from 6. The one left sits in code its profile never names. verdict-backfill: a repro for each old proven finding. Findings filed before repro blocks carry
prose only. The local model drafts a script with exactly one assertion markedDEFECT:, shown the
cited code with each file's import path and outline, with one repair round on a crash before the
assertion. Drafts that reach for a network or process module, call a destructive function, raise
the mark or mark more than one assertion are refused. Kept drafts run on a fresh checkout of the
record's last tested commit, then at the anchored commit: still broken (the next finalize attaches
the repro, origin marked), looks fixed (said on the gate'sbackfill:line, never resolved), or
no working repro.backfill.jsonlkeeps each row. A model-written repro never closes its
finding: when it stops reproducing,fix_suspectednames it for a run with judgment. The model is
shown how the project's own tests set up (Django settings, the head of the nearest test file); a
finding gets up to three drafts, each told why the last failed; a draft must touch the cited code,
a kept repro must run the cited function's body (a sandboxed tracer checks), and its one marked
assertion states what must not happen, so any correct fix passes it.eval/backfill/faithful.py
scores faithfulness against known fixes (a faithful repro stops reproducing once the fix is
applied): on five defects, the first drafts kept 0 of 5; as shipped, 5 of 10 drafts kept over two
trials, 4 of them faithful. On a copy of one owner's record (19 old findings, zero Claude tokens),
two trials kept 4 and 3 repros; read by hand, 3 of the 7 test their finding and 4 test something
near it, which is why such a repro never closes one. A finding left without a repro keeps what
its last draft ran into (last_why), so a miss can be told from a fix.- A model-written repro runs only in a sandbox, wherever it runs: the backfill, finalize's replay,
the nightly's in-run replay,verdict-replay. No network, localhost included; no writes outside its
fresh checkout (macOSsandbox-exec); a scrubbed environment and a scratchHOME. With no sandbox it
is not run. That is what lets it run where the profile saysprobes: off. - A session stopped at its dollar cap is not retried (any asked run): a second attempt would spend
the cap again. Nor is a run that wrote its state: a session-limit notice printed after finalize
used to cost an hour's wait and a second paid session. leads.json: every local run writes its leads as data beside the report.- Every profile key the harness reads is known (#164):
repro_rule,contract_rule,risk_paths
andassertion_gapswere announced as unread on every run. - A repro written
env A=1 python script.pyruns unattended, the same asA=1 python script.py;
it used to be refused over the wordenv, and a proven Major went unreplayed.env -istill is. - Pinned mutants: 36 new (escalate 1–12, backfill 1–10, focus 1–11, runner 1, replay 1, 0.91.2 profile 1); badge 493/493.
v0.91.1 — is the effect checked at all, and is the census ever cut short
- The census asks whether an effect is checked. Whole-statement mutants come first on a
line that is one: a call made for its effect, an attribute set and a raise becomepass, a
returned value becomesNone— is the save, the status change, the field written checked at
all? A mutant that does not parse is never run. 16 functions; money logic before admin UI
(dashboard/,admin/,templatetags/,migrations/last); a 480 s default budget. - A failing clean-tree set no longer hides a function. When the selected tests do not pass
on the fresh tree, each is tried alone and the passing ones kept; a skipped function says why. - The census is never cut short by the tester's Bash call.
verdict-runruns
verdict-factsbefore the tester starts, outside any Claude Code session, with the census's
whole budget; the tester's ownverdict-factsadopts those facts for the same HEAD and range
and claims the run marker for its session.--no-premeasurerestores the old order. Inside a
tester's Bash call the census takes what is left of a 9-minute window. - On by default for asked runs, unless the profile says
assertion_gaps: off; finalize
accounts for each survivor function — cited by a finding or named innot_tested— and under
repro_rule: onrefuses the run until it does. - The local tier uses the census. It ran none (it collects its own facts and never passed
the setting on); now it does, reads survivor files and functions first, hands the survivor to
its first question, and names unread survivors innot_tested. repro_rulebinds the agent's run, and the findings it filed. The local engine proves by
counterfactual, writes no repro and runs unattended; it is told, never refused, and so is a
model-free sweep. A finding carried from an earlier run is named, not refused: a record older
than the repro block holds proven findings without one, and turning the rule on must not stop
every later run for them.- A coverage render that exceeds 300 s falls back to the production files, so a large suite
keeps its reading map and census. - Measured at zero model tokens on stranger repositories: survivors 5 -> 18 on a thin-suite
library; on a large framework, ranked money-first, all 8 survivors in the money core; the local
tier with the census reads those functions first and files hypotheses there. - Pinned mutants: 17 new (0.91.1 gaps 5–10, local 1–5, premeasure 1–3, render 1, repro 1–2); badge 457/457.
v0.91.0 — the verdict means something
What a run claims is now checked by the harness, and where a run looks is ranked for it. Every
item below was measured on the October stranger records before it shipped; two candidates were
measured and dropped.
- The recall benchmark,
eval/recall/. A model-free scorer of what a run located, proved
(aprovenfinding whosereproreproduced) and blocked against known defects —
hand-verified stranger findings, money-key instances, clean changes — one finding per defect,
priced fromusage.jsonl.build_cases.pyderives a private case file from records. The
before column on six October records: located 17/17, machine-proven 0/17, blocked 2/17. - The repro is measured inside the run.
verdict-factsruns every open Major+ finding's
reproon fresh worktrees at HEAD and at the previous commit;verdict-finalizeruns the ones
filed this run at HEAD. A reproduction at HEAD refuses a resolution, claimed or by silence; one
that stops between the commits verifies a fix and resolves a finding nobody re-reported
(resolved_by: measurement (repro),fix_verified, outcomeconfirmedon a measured basis); a
new finding filedprovenwhose repro does not reproduce is refused with the failing check.
repro_rule: onrefuses a proven Major+ without arepro. Unattended, the harness runs argv
only — an interpreter on a QA-root or fresh-tree script,-c,-m pytest, orpytest— never
a shell pipeline: the command is the tester's and runs outside the Bash guard. 600 s budget. read_first. facts.json ranks where a passing suite proves least: changed functions by
changed lines no test executes, functions no test ever entered on a baseline
(reading_map.functions_never_entered), money and auth paths first (profilerisk_paths
globs, or whole-word / 4+-letter name stems). The prompt starts there and names what it did not
reach; the state carries which ranked functions a finding cited,not_testednamed, or nobody
accounted for, and the report renders a Read first table.- Assertion gaps (
assertion_gaps: on). On the coverage run's data, mechanical mutants in
risk-path functions that tests execute go into a fresh worktree one at a time, against only the
tests whose coverage contexts executed the line; a survivor is a line those tests run and do not
check, and leadsread_first. A clean-tree control and a canary per function keep an import of
the original checkout from passing for a gap. Measured on two stranger repositories: 20 s and 5
survivors on one with a thin suite, 164 s and none on one with a strong suite. verdict-check. A second look at a finished run on the local tier: which other sites share
each Major+ defect — sites fromgit grepon the cited line's identifier, outside the defect's
own function; the model only classifies. Leads toreviews.jsonl, never the state. ~1¢ a run;
on six records 4 leads, 1 a site the record's class listing missed.- Measured and dropped: a severity floor by risk path (on the Sales record it would have
raised 5 proven Minors to Major and corrected none); a severity question inverdict-check
(13 of 18 flagged, every one an upgrade along the standard's literal "financial error →
Critical" line). - Pinned mutants: 20 new (0.91.0 replay-in-run 1–7, read-first 1–5, check 1–4, gaps 1–4);
"scoped gate 5" re-anchored; badge 440/440.
v0.90.12 — a finding is a command a stranger can run
reproon a finding,verdict-replayto run it. A proven Major+ finding carries
repro: {command, cwd?, expect?}— the one command that shows the defect and what it
produces (an exit code, a substring on stdout or stderr; the symptom, never the fix).
verdict-replay <key> [ID …]checks the repository out again as a detached worktree at
HEAD, runs each command there with the fresh tree first onPYTHONPATH(an editable
install's.pthnames the original checkout and would quietly test it), bytecode and lazy
fetches off, any spelling of the original path rewritten to the fresh tree, and compares.
Results go to<qa-root>/replays.jsonl, outside the signed row; exit 0 every finding
reproduced, 1 one did not, 2 nothing could be replayed.--showprints the commands
without running one (they run in a shell with your rights),--keepleaves the worktree,
--timeoutis per command. Why: the pass that made the 2026-10-16/17 stranger reports
credible — a fresh clone, the finding's script, the expected value — was done by hand six
times and never appeared in a report built on the tester's own scratch.- The fresh tree is what runs. It comes first on
PYTHONPATH, and asitecustomizethe
command inherits disarms, after site-packages and before the command, an editable install
of the original checkout — a.pthline naming it, or a meta-path finder carrying its
build directory (meson-python; numpy-financial's replay said "reproduced" with every import
from the tester's own tree) — keeping the interpreter's own prefix. A path into the original
that the fresh tree lacks (the checkout's.venv/bin/python) stays where it is. Output that
still cites the original marks the rowisolated: falseand the gate saysNOT isolated.
A claimed reproduction that ended in a traceback whileexpectnames no stderr text, and a
fresh tree that cannot be imported (a compiled package with no build), arecould not run,
not a reproduction and not a refutation; a compiled package carries its build step in the
command ({repo},{qa_root}are the fresh tree and the QA root). - Measured on the six 2026-10-16/17 stranger records (scratch copies, the hand-run scripts
asreproblocks): 9 of 9 hand-verified Major+ findings on five pure-Python projects
reproduced on detached worktrees in 20 s; numpy-financial's three needed the build step. verdict-dupes: the tracker search a report should never skip. For each open Major+
finding, up to six single-word queries (the code names in its title, its words, what its
evidence cites) throughgh search issues --include-prs, ranked by the weight of the terms
that returned each item (title 1, evidence ½), a number or special value both titles share
(2½) and title overlap; the top five go to<qa-root>/related.jsonl, and the gate
names each finding's first lead (tracker #2381 open PR) and warns when it is an open PR.
A lead, never a resolution. Two of nine October stranger findings were already fixed by open
PRs, found by hand.- The gate reads the replay.
verdict-gatemarks each open findingreplay ✓ reproduced,✗ NOT reproducedor? could not runfrom the latest row per finding, adds
areplay:line with the tally and the proven Major+ findings nobody replayed, and carries
replaysin the JSON output. A record nobody replayed shows nothing. - The validator refuses a malformed
repro(not an object, an empty command, acwd
outside the checkout, anexpectwith unknown keys or a boolean exit). Absent is allowed:
verdict-finalizesays once which proven Major+ findings have none — "proven to its
author and to nobody else". - The finding template and the agent prompt carry the block;
docs/state-schema.md
documentsfindings[].reproandreplays.jsonl. - Pinned mutants: 16 new (0.90.12 replay 1–12, dupes 1–4); badge 420/420.
v0.90.11 — the project's tests are evidence of intent
A Critical that the project's own tests contradict is a question, not a blocker. On 2026-10-17
a baseline of a 43k-line shop framework said fail on a Critical — four discount types never
consume their condition's items — that 22 green tests assert the other way and the tracker had
tuned on purpose. The run knew: its own counterfactual fix broke those tests, and it wrote the
number into free text. Six stranger runs in two days showed the same shape twice.
- A fix that fails green tests is a contract dispute. A finding carries
fix_breaks_tests,
the count of tests that pass at HEAD and fail under the counterfactual fix; the prompt asks for
it and the validator enforces a count. Above 0, finalize marks the findingcontract_dispute,
caps Blocker/Critical at Major (severity_waskeeps the filed level), takes it out of the
release blockers and parks a question for the maintainer with three answers. Afailthat
rested on it alone becomespass with risks, recorded asverdict_rule: contract dispute.
Inside a money or security gate's own range the money rule keeps precedence.contract_rule: offin the profile disables it. - Two isolation slips the session now stops.
verdict-runcarries
PYTHONDONTWRITEBYTECODE=1(a tester's barepythonwrote 113.pycfiles into a checkout)
andGIT_NO_LAZY_FETCH=1(git log -Son ablob:noneclone fetched 22 MB from GitHub) into
the tester's environment. - Pinned rules 395 → 404, each killed by the suite.
Not in this release: a hand-run verdict-finalize inside a live session reads that
session's transcript as the tester's bill, so a session that changed model reads as a
two-model tester (blocked); the bill should bind to the run's own session. The certainty
pass — a fresh-checkout replay of a finding's repro, and a duplicate search in the target's
tracker — is designed and not built.
v0.90.10 — what changed, and what only a commit claims
A standing verdict is not news; a new regression is. Sales stood at fail for 18 runs on two
items that wait on a person — a decision and one production command — and the nightly, which pages
only when a night itself breaks, said nothing for weeks. It would have said nothing about a new
regression either: both nights exit 1. And 8 of its 73 open findings had a later commit on main
naming them in its subject, carried open for weeks.
verdict-runandverdict-gatesay what the last run changed. News is a verdict that moved
against the run before it, or a finding at Major or worse that is new, came back, or was resolved.
One line:news — …, orno news — the verdict stands at 'fail' · waiting on a person: … · blockers: …; a night that ran nothing says so instead of repeating the last run's news. Release
blockers a parked question has handed to a person are listed apart from the rest. Exit codes are
unchanged;--format jsoncarriesnews(anyis the bit to page on).- A fix somebody claimed is named, never assumed. The gate lists
fix_claims: open findings
that a commit in the measured history names in its subject (census: … (F-113) (#330)), dated on
or after the day the finding was first seen, touching more than the QA record. The standing line
ends "a commit names as fixed, no run has verified: …". A claim is a lead for the next run, not a
resolution: two of Sales' eight are rightly still open. - The skip scan reads the tests the suite collects. It walked every
test_*.pyin the tree, so
it filed an eval fixture pytest never collects as an abandoned skip on every run of Verdict's own
repository. It now reads the test-id ledgerverdict-factswrites, and walks the tree only
without one. Apytest.skip()call under anif,exceptor loop is a guard, likeskipif. - The Bash guard knows the command's own assignments.
S=.qa/scratch; mkdir -p $Swas refused
as an unresolved variable: 21 of the 33 hook refusals in the 2026-10-15 trace. A value is known
only when it expands to a literal;FOO=x cmdand a pipeline stage set nothing for later commands. - Pinned rules 377 → 395, each killed by the suite.
Not in this release: the local tier still never lifts a standing fail; a first run with no
profile whose guessed interpreter lacks the project's dependencies still reads the collection errors
as a red suite; the Claude tier does not yet record which test design technique a finding came from.
v0.90.9 — the model's time is the model's
A release gate read nothing and said the model had run out of time. The local tier's wall-clock
ceiling (--max-model-s, 900 s for a gate) started when its budget was built, before the suite and
coverage ran. On Verdict's own repository the three release gates of 2026-10-15 spent 663, 716 and
853 s of their 900 s measuring, and read 24, 5 and 0 functions; the last one reported "the 900s
model budget ran out" without a single model call.
- The ceiling is model time. Its clock starts at the first question to the model; the suite and
coverage before it are measurement. Over the same range, the fixed engine read 24 functions (the
cap) where the released one read none. Nights mostly escaped the defect: their ceiling is
3,600 s, and since 0.90.7 they reuse the sweep's measurement. finding_of's docstring said every local finding is a hypothesis, which stopped being true when
counterfactual probes arrived; the fixed release gate, reading for the first time, filed the stale
sentence as a proven Critical. It now says what the code does.- Pinned rules 376 → 377, each killed by the suite.
Not in this release: changed-line coverage on a branch gate is measured only when the gate is
given its range (verdict-run --range); the local tier still never lifts a standing fail; a first
run with no profile whose guessed interpreter lacks the project's dependencies still reads the
collection errors as a red suite.
v0.90.8 — the tester is the session
An asked run could start without Verdict. On 2026-10-06 four asked money gates ran from a
packet that never asked for the agent: a generic session at effort medium, without Verdict's
prompt, and two of the four ended their turn with no state. verdict-run only asked for the tester
in words, and the rule that catches a run never finalized only read a subagent's stop.
verdict-runlaunches the tester as the session:claude --agent verdict --effort <the effort its agent file pins>, with a prompt for a session that is the tester. The operator's own
prompt,--agentor--effortis kept. A CLI that refuses the flag (before any model call) is
launched the old way, and the refusal is not an attempt; with no agent file nothing changes.- The never-finalized rule sees that session's Stop. A session started as an agent names it on
its own hook events (measured on CLI 2.1.281); a plain session's Stop names none. So a tester that
ends its turn withoutverdict-finalizeis told once whether it ran as a subagent or as the
session, and the main session ending its turn beside a background tester still says nothing. - A gate in a root of its own says it was a gate. Every branch gate runs in a throwaway root,
and every one was reported as "run 1 (baseline)", as if the whole repository had been read. With
a range and no previous state,verdict-factsnow labels the run "Gate over BASE..HEAD, in a root
of its own: the change is the scope, not a baseline of the repository";run_typestays
baseline, the strict enum. - The SWE-bench control arm still launches plain Claude Code, with neither
--agentnor--effort. - Pinned rules 365 → 376, each killed by the suite.
Not in this release: changed-line coverage on a branch gate is measured only when the gate is
given its range (verdict-run --range); the local tier still never lifts a standing fail; a first
run with no profile whose guessed interpreter lacks the project's dependencies still reads the
collection errors as a red suite.