Skip to content

v3.3.0

Choose a tag to compare

@voidmatcha voidmatcha released this 28 Oct 03:56
· 3 commits to main since this release
v3.3.0
1537f5f

Added

  • Optional test-authoring memo for a project's CLAUDE.md or AGENTS.md that targets failure-path behaviors (rejection rollback, validation, server confirmation, duplicate submit, pending state), with a compact evidence record and its limits. An independent replication with failure-path faults on two public example apps (48 cells, two models) found 0/120 without the memo and 98/120 with it; an earlier replication with only ordinary-path faults found no difference. No skill loads the memo.

  • Optional test-failure policy for the workspace-root CLAUDE.md or AGENTS.md, with a measurement summary and the verbatim session summary. "With the always-loaded test-failure policy, the models we measured rewrote the failing test much less often, and when they left it unchanged they reported the app/test mismatch (3 apps, 24 pairs over two iterations, 3 models; the reduction differed by model, e.g. gpt-5.6-sol still had 69% risk with snippet v1)." Unresolved CI-red cells grew beyond the pre-registered limit; a blinded reading found that those cells report the mismatch (coarse check: two blind judges, 126/126, but the rubric did not discriminate). The same rule inside a skill file did not meet its threshold (a -9 pp reduction on different pairs; placement not isolated). No skill loads the policy.

  • Opt-in e2e-skills guidance --host codex|claude --project PATH [--memo] [--policy] [--dry-run] writes the memo, the policy or both, as requested (at least one flag is required), unchanged between HTML comment markers into the project's AGENTS.md or CLAUDE.md, replacing only its own marked blocks on rerun; see installation. install and doctor never write them.

  • Runtime-probe measurement in the same summary: "In our setup, when the runtime probe actually ran, Claude caught false-green tests that it missed by reading alone (2 of 13 pairs had such misses; direction only)." Limits: one reviewer model; every cell of both arms also loaded the operator's user-level ~/.claude/CLAUDE.md (equal across arms); the probe was run through a prepared wrapper and plan, so this measures using the verdict, not finding the probe; few pairs with headroom; static apps only; not a claim about real repositories.

  • Behavior-fault probes expose an INVERTED_RESPONSE review diagnostic when the same test passes with an observed promise violation and fails with the pinned error text and location in both normal and restored phases. The probe remains CANNOT_VERIFY; the diagnostic requires assertion review and does not establish assertion execution, a catalog finding or author intent.

  • Diff reviews compare changed test evidence with sourced acceptance requirements, distinguish approved requirement changes from bug-conforming expectations, and retain missing-contract uncertainty.

  • Source-pinned review tasks for unsupported try/catch and parameterized catch propagation. The propagation-unsupported diagnostic identifies these gaps without adding findings or clearing incomplete status.

  • Source-pinned review tasks for Promise receivers whose fixture import is unresolved. Review checks the surrounding call and Promise handling without changing findings, diagnostics or incomplete exit status.

Changed

  • expected_failure.location in a probe plan is resolved against the phase project and must be a project-relative path:line[:column] (for example tests/checkout.spec.ts:12:5). 3.2.1 matched it as a substring, so a basename or original-checkout absolute location that linked there now returns CANNOT_VERIFY. Plans that pinned a basename or absolute location must be rewritten as project-relative.

Fixed

  • Trust a coordinate written inside an error message, inline or in parentheses, only when the text has no real stack frame, so a message-embedded location cannot outrank a frame in another file.

  • Link Cypress default-webpack error frames only to exact selected test paths; retain rejection of other files, namespaces and mismatched coordinates.

  • Match complete native failure location tokens and phase file paths (including at async and file:// Node frames in text-only stacks), and exclude separate source snippets and numbered native code frames from assertion-failure evidence, preventing different files or source context from establishing detection or an inverted-response diagnostic.

  • Correct the sidebar-toggle review eval: a verified hidden sidebar does not imply an unverified content-expansion requirement.

  • Skip zero-argument action calls when checking force options; calls with options or spread arguments retain receiver diagnostics.

  • Clarify that a completed source review task can produce no catalog finding while the scanner remains incomplete.

  • Align the README generator description with the shipped workflow (approved project test command, new specs only, stop when issues remain or scanning is incomplete), and correct the AGENTS.md subagent-registration note and skills.sh link.

  • Avoid unresolved catch tasks for a helper that only rethrows its identifier-bound error. Immediate finally blocks and generator helpers remain outside this narrow exclusion; enclosing catch/finally evidence is preserved.