Releases: akahkhanna/groundtruth
Release list
v2.1.0
Three additive, backward-compatible honesty improvements over 2.0.0. No config change
needed; opt out of the whole contract engine with GROUNDTRUTH_CONTRACT=0 as before.
Added — multi-turn deferral persistence (spec §6)
A deferred claim declared on an earlier turn now stays visible on later turns' cards
until verified work closes it, reconstructed from the transcript (the unforgeable record)
rather than a forgeable ledger — so a declared set-aside can't silently vanish by being
omitted next turn. Closure uses only verify-grounded channels and a distinctive key.
Added — prose class-1 fallback for contract-unaware turns (warn-only)
A session whose agent never emits a groundtruth-claims block now still gets a bare prose
"tests pass" lie caught (warn), reinstating v1's _passClaim shape (a test/build noun +
a pass-verdict verb, with negation/modal/reported-speech guards). Never fires when a valid
block exists, abstains with no transcript, and doesn't fire on honest closings like
"Done — fixed the typo."
Changed — no_change contradiction is now block-eligible
A no_change claim may no longer coexist with any other claim (schema-invalid → NC), and
a sole no_change claim on a turn that authored non-excluded changes is now a block-tier
CA self-contradiction (naming up to 3 paths), with the redundant per-file UC warns
suppressed. An empty authored set or an all-excluded diff stays silent (the honest case).
Added — tree-state stamping for tests_pass/build_pass staleness
The green-run staleness check previously relied on within-session event ordering (the
last green run's position vs the last source edit). Its blind spot, surfaced by
community review: an edit made through the bash channel (sed -i, a heredoc,
> f) never updates the edit position — it tracks only Write/Edit tool ops — so a
green predating such an edit passed clean.
This is closed by recording the repo's code fingerprint independently at command time
and comparing it at claim time — grounded in code state rather than event ordering, so it
sees a bash-channel edit directly:
- Stamp — a new
PostToolUse[Bash]hook (--stamp) records{ head, tree }after
everyTEST_BUILD_REcommand, appending one JSON line to
.claude/groundtruth/<session>-stamps.jsonl.treeis a hash of the repo's normalized
code — theCODE_EXT_REsource files (comments and whitespace stripped, the same scope
and normalization the event-ordering anchor already treats as "staling"), read straight
from disk. So it does not fire on a docs/config edit, a comment-only or whitespace-only
edit, a harness-written settings file, or a commit of the tested code (the fingerprint is
absolute content — HEAD identity is not compared); a real code change through any channel
does. Groundtruth's own.claude/groundtruth/state is excluded. Non-mutating git only
(--no-optional-locks); fail-open (not a repo / git error / non-matching command → writes
nothing). - Sensor — a
stale-by-treecheck runs ahead of the event-ordering check in the
at-most-one-fires chain: when the stamp for the last matching green run differs in code
fingerprint from the value computed now, it emits a warn-tierCA("the working tree has
changed since that run — the green is STALE"). Every uncertainty abstains (no stamps,
malformed line, ambiguous stamp↔run mapping, failed current-tree computation), falling back
to the unchanged event-ordering check — a false "stale" on an honest green is the fatal FP
this must never emit. Because the stamp log is agent-writable on-disk state, the sensor can
only add a stale warn; afreshverdict never suppresses the transcript-anchored
event-ordering warn. - Tamper surface — the stamp log is in the referee-tamper perimeter: an agent-authored
Write/Editto it is flagged. A deleted or missing log is treated as absence (abstain),
never as evidence.
Scope note: stamps are per-session (<session>-stamps.jsonl); a claim citing a command
with no matching run in this session is already intercepted by the "no such command ran
this session" check, so the sensor's reach is within a session (including a resumed one).
Design credit to community review for identifying the blind spot and the
record-at-command-time / compare-at-claim-time approach, and for the review pass that
rescoped the fingerprint from the whole tree to normalized code (avoiding a false-stale on
every honest post-green docs/comment/commit turn).
Full symptom → root-cause → fix → regression-test in FIXES.md.
v1.4.0 — Class-3 content grounding (admin.html FP), B4 unscoped UPDATE/DELETE warn (incl. tautology WHERE), MCP-SQL scanner hole closed
-
LIVE FP (hindsight f7df5ae9, same-day report): Class 3 flagged "claimed
a change to admin.html, but it is absent from the diff" when the agent
edited CLAUDE.md's MENTION of admin.html and said so. Class 3 grounded
claim-named files against changed PATHS only; it now also grounds on the
diff's ± content lines (the ledger's grounds() has since Phase 6), with
boundaries so site-admin.html can't ground admin.html while
public/admin.html does. Review found + fixed the col-0 deletion-marker
asymmetry ('-admin.html' failed to ground while '+admin.html' grounded). -
New B4 (WARN only, user request): UPDATE/DELETE whose WHERE is absent
OR a tautology (1=1, 0=0 OR 1=1, true, 'x'='x') — B3's constant-predicate
class applied to writes. Fires only on a complete;-terminated statement
whose head anchors UPDATE/DELETE FROM; incomplete tails, CTE heads,
comments, ON CONFLICT DO UPDATE, prose all abstain. Identifiers are
\p{L} (unquoted non-ASCII columns scope) and $n params scope. -
FN (silent-inertness): the pseudo-file — captured expressly
so MCP-applied SQL reaches the security scanners — never matched B1/B3's
\.sql$; an apply_migration CREATE TABLE with no RLS produced ZERO
findings. Matcher fixed; opening the channel exposed a block-tier FP
(read-only SELECT quoting DDL in a string literal fired B1/B3), closed at
root by blanking single-quoted literals before every SQL check — which
also fixed B4's literal-;FP and literal-"where" FN.
Adversarial review (Fable), three passes: marker asymmetry; the SELECT
block-tier FP + literal blanking; the ASCII-only identifier hole. Each
CHANGES REQUIRED → fixed at root → re-verified → APPROVED for the batch.
Self-check 726 → 763; red-team 14/14; corpus unchanged (17/23).
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.4.0).
v1.3.2 — three live FPs from one field report: bystander rule-fire, immortal ledger nags, diagnosis-only asks
-
A compiled rule fired 🔴 on an untracked BYSTANDER file: the Stop path
fed runCompiledRules the wider scanDiff, whose untracked-content merge
exists for the SECURITY scanners — so no-commit-local-only-files matched
the shebang of scripts/dev-server.mjs, a file never committed, staged, or
edited that session. Rules now grade the AUTHORED reality only (git diff +
tool ledger + MCP SQL); documented trade: a Bash-heredoc-authored file
dodges doc rules in-session (security scanners + pre-commit/CI still see
it). End-to-end regression through the real Stop path, both directions. -
The task ledger never healed: tier was backfilled only when null, so a
task minted HARD by a PRE-v1.2.1 classifier (t1d7d, "I do have image
attirbutions.md…") nagged for 300+ turns after the classifier was fixed.
tier + deliverable are now RE-DERIVED every turn (idempotent) and an ask
yielding no deliverables is reaped — every classifier fix is now
retroactive. Reap is TWO-PHASE (minted keys win) after review found the
100-char truncation-key collision could silently delete another ask's
live task, order-dependently. -
A DIAGNOSIS-ONLY ask minted unclosable HARD deliverables (t7dep,
"…Game.js has been modified heavily… Can you see what is happening"):
passive-voice REQUEST verbs read as commands, and the correct outcome was
an EMPTY diff (the agent restored the file). New ask-wide diagnosisOnly
gate: no active REQUEST verb (passive phrases + determiner-preceded
verb-nouns blanked) AND a question/read/verify clause → all tokens soft.
The 4th instance of the v1.0.5/v1.2.1/v1.3.1 unclosable-wedge root defect.
Adversarial review (Fable): CHANGES REQUIRED → fixed → APPROVED. Findings:
2^n backtracking in PASSIVE_VOICE_RE ('recently' vs \w+ly ambiguity, 57 s
at 28 reps — would wedge every Stop after one poisoned paste); the reap
collision above; gate-boundary FN (EXTRA_REQUEST_VERB_RE, scoped to the
gate's off-switch so it can only restore pre-fix behavior, never mint).
Healing verified against the REAL hindsight ledger: t1d7d + t7dep re-derive
to soft and stale out silently; tzjeb (genuinely deferred work) stays
pending. Self-check 702 → 726; red-team 14/14; corpus unchanged (17/23).
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.3.2).
v1.3.1 — agent integrity + three live FP fixes
New check (agentFindings, pure core + collectAgents fs wrapper; wired into
--audit and SessionStart, the only moment subagent definitions are read).
THE INCIDENT (hindsight, verified via git): 16 subagents lived in
hindsight-vercel/.claude/agents/ — one level BELOW the repo root. Claude Code
resolves .claude/agents/ by walking from the CWD upward and never descends, so
launched from the repo root not ONE ever loaded — silently, for weeks — while
CLAUDE.md claimed migrations were "enforced by the auto-invoked
migration-reviewer subagent". An agent that cannot load fails OPEN: no error, no
log. That is "fail-loud on silent-inertness" (CLAUDE.md) applied to subagents,
and a doc resting on one is Class 4 (phantom ref) over a new artifact.
It flags, all as STATIC file properties: an agents dir below the repo root;
missing frontmatter / name / description; a DOTTED model id (claude-opus-4.8 ->
claude-opus-4-8) — deliberately NOT a model allowlist, which would go stale and
false-fire on the next model shipped; duplicate names and byte-identical
descriptions (the router selects on description); and a phantom agent reference.
SCOPE, deliberately: it proves an agent CANNOT fire. It never claims one WILL —
selection is the model's discretion, and claiming otherwise is the overclaim the
positioning guard forbids. All five real failure modes lived in "cannot fire".
Replay: 14 findings at 8b93544^, 0 at HEAD.
Three live FPs from the same sessions, fixed:
- vacuous-test: blanking strings BEFORE stripping comments let an apostrophe in
prose ("// … France's longitude span …") open a phantom string that swallowed
a test body's braces AND itsawait dc.checkRound(...)— the block extracted
EMPTY and a test that plainly asserted was reported as asserting nothing. One
leftmost-first string|comment|regex pass. - compiled rules: comments were stripped only for CALL rules, so an IDENTIFIER
rule (import.meta) fired on the very COMMENT explaining it. Default inverted:
code-only unless the rule TARGETS comments — so nothing goes inert. - ledger: a VERIFICATION ask ("verify game.js does not need updating") minted an
unclosable HARD deliverable and escalated to BLOCK on work done correctly — an
empty diff there is SUCCESS. Third instance of one root defect (v1.0.5 pasted
listing, v1.2.1 declarative): the ledger presumed every named file must land in
the diff. Demoted to soft, demote-never-drop.
Adversarial review (Fable, three passes) found 13 real defects across the above,
every one fixed at root and pinned — incl. an @-in-pattern classifier that
condemned every decorator/npm-scope rule and RE-OPENED the prose FP it fixed;
untracked agent files invisible to git ls-files (the most likely home of a
broken agent — all four new ones were untracked); a bare-word phantom that
missed the incident's second blocker ("invoke the reviewer subagent"); two FNs
in the verify demotion (cross-clause anaphora, and a verify verb hiding INSIDE
the filename); a case-fold that invented a collision between QALead and qalead;
and a broken agent file minting a phantom about itself.
Self-check 638 -> 702; red-team 14/14; corpus unchanged (17/23 at-capture).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.3.1).
v1.2.2 — failure sensor scoped to the run backing the claim
Class-1's failed sensor scanned EVERY result in the session for TEST_FAIL_RE,
so it fired on:
- a red run that was then FIXED and re-run GREEN — the stale failure text
lingers inresultsforever, so the normal red->fix->green flow warned on
every subsequent turn; - ANY output merely CONTAINING "FAIL"/"N failed" — a probe script printing a
**FAIL**marker, a quoted CI log, a pasted report.
Measured as the single noisiest finding in GT's own development: ~40 fires in one
session, nearly all false.
Fix: bind failed to lastRun — the last completed, non-background,
non-aborted family run, i.e. the same run the v1.1.0/v1.1.1 exit-status sensor
judges. Both sensors are kept deliberately: exit catches a crash with no failure
string ("Killed"); the substring sensor catches a runner that prints "3 failed"
and still exits 0. With NO bashEvents (--pre-commit / --diff-range / legacy
transcripts) the old session-wide scan is kept verbatim — zero drift, proven by
running the committed pre-change suite (627/627) against the new engine.
Fable, on review — the scoping UNMASKED a pre-existing inversion: onlyFiltered
graded outcome-less command STRINGS, so an unfiltered run defeated the .every
even when it FAILED. A failing run therefore made the verdict CLEANER: run the
suite red, then a trivial --grep green, claim "all tests pass" -> silent. It was
masked only because the session-wide substring accident fired on the red's stale
text. Now the filtered check grades the GREEN completed runs when outcome data is
trustworthy (>=1 green completed family run, none unpaired/background — the same
abstain guards the other artifact sensors use), else the legacy command grade.
ROADMAP: Class-1 marked fixed. Class-9 self-match recorded as a DELIBERATE NO-FIX
with reasoning — EVAL_CODE_RE correctly flags a real branch on GROUNDTRUTH_* env
vars, which this engine genuinely has; the only sound discriminator is identity,
and a path-scoped self-exclusion would blind Class 9 to real special-casing inside
Groundtruth itself. The previously-proposed declaring-file precedent does not
apply (an env-var branch has no declaring file). Accept the noise.
Self-check 627 -> 638; red-team 14/14; corpus unchanged (17/23 at-capture).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.2.2).
v1.2.1 — declarative asks no longer mint unclosable blocking deliverables
Live FP from a real hindsight session: "I do have image_attributions.md on
downloads folder" — the user stating a file EXISTS — minted a HARD (blocking)
deliverable. The file lived in ~/Downloads, OUTSIDE the repo, so it could never
ground in a git diff, so the task could never close: it nagged every turn and
under block mode escalates to a BLOCK on work never requested and unable to
exist. Same phantom-deliverable / block-mode wedge v1.0.5 closed for pasted tool
listings, arriving through a different door.
Root cause: isTrackableRequest() has a !framed -> return true fast path, and a
declarative is neither a dismissal (NON_REQUEST_RE) nor a question (QUESTION_RE),
so it fell straight through to "plain imperative -> track".
Fix: DECLARATIVE_RE + a declarative gate in classifyDeliverables that DEMOTES
to soft (surfaced once, auto-expires, never blocks) — never drops, per the
demote-never-drop invariant.
The first cut of that fix was WRONG in the dangerous direction, and both FN
families are now closed:
- Cross-clause anaphora: splitClauses splits on
., so "I have a notes.md.
Add it to the repo." put the token in the declarative clause and the verb in
the next one — silently demoting a GENUINE request (the completeness backstop
going inert, the cardinal sin). Fixed with an ask-wide anaphoric-request scan. - Problem reports (Fable, on review — a family missed entirely): "we have a bug:
cache.js returns stale data", "there is no error handling in parser.js",
"a race condition exists in cache.js" are declarative in FORM but a commission
in FUNCTION. Fixed with a PROBLEM_RE negative gate.
Both gates can only RESTORE pre-fix hardness, never mint new hardness — zero new
FP exposure; worst-case over-match degrades to the old behavior.
Self-check 613 -> 627; red-team 14/14; corpus unchanged (17/23 at-capture).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.2.1).
v1.2.0 — vacuous-test warn + verification-tier taxonomy
New Class-1 check vacuousTestFindings (warn-only, claim-gated, JS/TS only): an
added test whose body makes no call and does not throw provably cannot fail.
Scoped to be false-positive-free rather than broad — a delegated assertion, a
table-driven loop, and a "doesn't throw" smoke test all have a call and stay
silent; a genuinely empty/console-only body fires. Fires only on a fully-added,
brace-balanced block (edited-test bodies outside the hunk abstain).
Adversarial review (Fable) found four FP families in the first cut — a gap-blind
line join, unhandled regex literals, and the thesis hole that chai .should
getter chains and bare await are call-free failures — all fixed at root and
pinned; .should/await now count as actions, and a gap sentinel enforces the
fully-added-block invariant.
ROADMAP.md: the "deterministic vs needs-a-model" line is split into four
verification tiers (T1 artifact-held / T1.5 execution-evidence-in-transcript /
T2 empirical / T3 model), prompted by a reader who caught a real misfiling —
9d passes-only-the-visible-test and 8 regression-blind are T2 (a test run), not
the model layer. Also logs two known self-match FP families (Class-1 substring
scope, Class-9 own-source) as tracked deferred precision.
Self-check 596 -> 613; red-team 14/14; corpus unchanged (17/23 at-capture).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.2.0).
v1.1.1 — orchestrator FP + exit-attribution + inert-edit fixes for the Class-1 evidence layer
Three fixes, each surfaced by putting v1.1.0 through two Fable passes (a
break-it red-team + an architectural review) and a real subagent transcript:
-
SubagentStop falsely accused every test-running subagent of "no test ran":
the sidechain filter (correct for the main path) deleted the subagent's own
evidence while keeping its claim, and the stop also wrote shared session
state against the ORCHESTRATOR's contract. main() now branches on
hook_event_name===SubagentStop: parse the subagent's own file with
includeSidechain, grade its claim against its own evidence, write nothing
shared, demote block->warn. Empty/unreadable transcript -> abstain. -
Exit-status attribution was wrong across every shell connector:
&& echo ok
laundered a crash,cd x && npm testfired on a test that never ran,
|| true/pipe/;laundered a red into a green. Replaced with a
connector-aware rule (|| -> null; exit belongs to the last;/newline
statement; within the last statement attributable iff every non-family
segment is infallible {echo,true,:}; a null latest run abstains and never
displaces an earlier red). Command-position anchor (FAM_CMD_RE) so a runner
name in an argument (grep vitest ...) is not a family run. -
Stale-green fired on inert edits (a comment on a code line, whitespace
reformat): the mutation-is-code test is now the normalized code portion
(splitCodeComment + collapsed whitespace), so comment/whitespace deltas
don't count while string/code changes still do.
The review found five live defects in the half-finished implementation, two of
them the exact laundering/FP classes the changelog claimed closed (trailing
;/newline laundering, lone-& FP, redirected-echo FP, runner-name-in-arg
FP/FN, empty-subagent-transcript re-mint) -- all fixed at root with pinned
regressions.
Self-check 566 -> 596; legacy committed suite 509/509 (zero drift);
red-team 14/14; corpus unchanged (17/23 at-capture).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.1.1).
v1.0.5 — the paste fix, actually delivered (1.0.4 shipped it under a published version number)
If you are on 1.0.4, you do not have the fix below — update. v1.0.4 shipped the paste fix into an already-published version number, so /plugin update compared 1.0.4 == 1.0.4 and delivered nothing. 1.0.5 is the first version that actually carries it. (A silent no-op of exactly the kind this tool exists to catch: real work, real commit, real release — and it still did nothing.)
509 self-checks · red-team 14/14.
A pasted tool listing minted a phantom HARD deliverable — and phantom deliverables escalate to block
An operator pasted an MCP-server status listing raw — Project MCPs (…/.mcp.json) ❯ supabase · ✔ connected — purely as context. The task ledger read .mcp.json / claude.json as requested deliverables, minted a pending task, and since no commit ever touches those files it could never close — so every subsequent "done" tripped false completion (async), forever.
Root cause: pasteStripped() recognized only fenced ``` blocks, >-quoted lines, and file.js:42 stack refs. A raw pasted status/tool-output block — the way people actually paste — read as prose, so its filenames became hard deliverables.
Why it matters now: an open loop escalates to block. With block mode on, every pasted tool-output block naming a file would wedge the agent on nothing — precisely the failure that makes an operator rip block back out.
Fix: pasteStripped() now also strips lines carrying a structural tool-output glyph (❯ ⎿ ⏺ │ ├ └ ┌ ┐ ═ ║ …), so their tokens become paste-refs → soft (surfaced once, auto-expires, never blocks, still grounds to done). Nothing is dropped — the demote-never-drop invariant holds. A whole-line strip is required rather than glyph-onward: the motivating listing has ❯ mid-line, so anchoring would leave .mcp.json hard and un-fix it.
Adversarial review (Fable) removed ▶/► from the glyph class — the only ones a human writes in prose (refactor parser.js ▶ tokenizer.js), which they wrongly demoted. Accepted trade, pinned by test: an imperative sharing a line with a real tool glyph (vitest prints ❯) demotes to soft; restating it on a glyph-free line restores hard. Known-narrow: glyph-less pastes (plain ls, ASCII | tables) still mint.
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.0.5).
v1.0.4 — throwaway-path noise (oversized tmp/ warn) + enforcement-boundary invariant
⚠️ Correction: this release originally also advertised the pasted-tool-listing / phantom-deliverable fix. It does not ship in an installed 1.0.4 — that fix went into the tree without a version bump, so/plugin updatesaw1.0.4 == 1.0.4and delivered nothing. It is released properly in v1.0.5. What 1.0.4 actually contains is below.
Two throwaway-path fixes from real sessions. 502 self-checks · red-team 14/14.
The oversized-file warn fired every turn on a throwaway tmp/ blob
A 3 MB untracked tmp/reconcile/diff.json — never touched, never committed — produced "untracked file too large to fully scan for secrets" on every single Stop for a whole session. untrackedAdded() skipped only the changed-file set and .claude/groundtruth/, not the throwaway paths (tmp/temp/scratch/…) that excludedScanPath() already knows about; the oversized list bypassed the downstream dropExcludedFiles() filter and leaked straight into findings. Now excluded at the source of the untracked loop — a throwaway file is neither read nor surfaced, and excluded files consume zero scan budget, so more real content gets scanned before the cap.
Not a security change: an excluded path's content was already never scanned via the Stop path, so removing the vestigial warn takes away no coverage.
Enforcement-boundary invariant pinned
The Stop path suppresses tmp/ noise, but the merge gate must not: pre-commit (--pre-commit) and CI (--diff-range) pass the raw diff to analyze(), so a committed tmp/ secret still blocks. That now rests on a silent invariant, so a regression test asserts analyze() still flags a secret on a tmp/ path — it fails loudly if a future refactor unifies the exclusion and takes committed throwaway-path secrets dark at the boundary.
Full symptom → root-cause → fix → regression-test in FIXES.md (v1.0.4).