Skip to content

v1.2.2 — failure sensor scoped to the run backing the claim

Choose a tag to compare

@akahkhanna akahkhanna released this 14 Jul 15:41
· 10 commits to main since this release

Class-1's failed sensor scanned EVERY result in the session for TEST_FAIL_RE,
so it fired on:

  • a red run that was then FIXED and re-run GREEN — the stale failure text
    lingers in results forever, so the normal red->fix->green flow warned on
    every subsequent turn;
  • ANY output merely CONTAINING "FAIL"/"N failed" — a probe script printing a
    **FAIL** marker, a quoted CI log, a pasted report.
    Measured as the single noisiest finding in GT's own development: ~40 fires in one
    session, nearly all false.

Fix: bind failed to lastRun — the last completed, non-background,
non-aborted family run, i.e. the same run the v1.1.0/v1.1.1 exit-status sensor
judges. Both sensors are kept deliberately: exit catches a crash with no failure
string ("Killed"); the substring sensor catches a runner that prints "3 failed"
and still exits 0. With NO bashEvents (--pre-commit / --diff-range / legacy
transcripts) the old session-wide scan is kept verbatim — zero drift, proven by
running the committed pre-change suite (627/627) against the new engine.

Fable, on review — the scoping UNMASKED a pre-existing inversion: onlyFiltered
graded outcome-less command STRINGS, so an unfiltered run defeated the .every
even when it FAILED. A failing run therefore made the verdict CLEANER: run the
suite red, then a trivial --grep green, claim "all tests pass" -> silent. It was
masked only because the session-wide substring accident fired on the red's stale
text. Now the filtered check grades the GREEN completed runs when outcome data is
trustworthy (>=1 green completed family run, none unpaired/background — the same
abstain guards the other artifact sensors use), else the legacy command grade.

ROADMAP: Class-1 marked fixed. Class-9 self-match recorded as a DELIBERATE NO-FIX
with reasoning — EVAL_CODE_RE correctly flags a real branch on GROUNDTRUTH_* env
vars, which this engine genuinely has; the only sound discriminator is identity,
and a path-scoped self-exclusion would blind Class 9 to real special-casing inside
Groundtruth itself. The previously-proposed declaring-file precedent does not
apply (an env-var branch has no declaring file). Accept the noise.

Self-check 627 -> 638; red-team 14/14; corpus unchanged (17/23 at-capture).

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Full symptom → root-cause → fix → regression-test in FIXES.md (v1.2.2).