v0.12.1 — Action renamed for GitHub Marketplace

The GitHub Action is renamed "Agent Roast: weakened-test guard" so that it can be listed on GitHub Marketplace: the plain name agent-roast belongs to a GitHub organization, and Marketplace rejects a name that matches a user or an organization.

uses: Nari-Ab/agent-roast@v1 is unchanged. The CLI is the same as in 0.12.0 (see the 0.12.0 notes for the agent hook, working-tree review, demo and the new languages).

Full changelog: https://github.com/Nari-Ab/agent-roast/blob/main/CHANGELOG.md

v0.12.0 — Send your coding agent back when it weakens a test

Until now agent-roast looked at a change after it was committed. This release looks at it while it
is still being made, where an agent can still fix it, and makes the first run show something useful.

Guard a coding agent.

• New agent-roast hook install adds a Stop hook to Claude Code, Codex and Cursor. When the agent
  is about to finish, everything it changed since HEAD is reviewed, and if a check got weaker the
  agent is sent back once with the file, the line and the before and after. A second attempt is not
  blocked again; the person gets a warning instead, so the two never loop. Anything that cannot be
  checked lets the agent finish. hook status, hook uninstall, hook pause and hook resume
  manage it, and the hook pins the version that installed it. --instructions also writes a short
  paragraph into AGENTS.md and CLAUDE.md.
• New Claude Code plugin and marketplace entry: /plugin install agent-roast --marketplace Nari-Ab/agent-roast.
• New review --worktree compares HEAD with everything not committed yet (staged, unstaged and
  untracked files). Git compares against a private copy of the index, so the repository is never
  touched; the policy still comes from HEAD. Context reviews report schemaVersion 1.2.0 and
  source: "worktree".

A first run that says something.

• agent-roast without arguments now reviews what you have not committed yet, and on a clean tree
  replays your recent commits. It used to print the history score, which moved to agent-roast audit
  (--all, --since, --base and the other old options still work as before). --json,
  --format, --fail-on and --advisory apply to the new default.
• New agent-roast demo shows a scripted example in a temporary repository, with the real review.
• The terminal review colors FAIL and WARN findings and the verdict.

More languages. test-skip, test-focus and trivial-assert now cover Go, Rust, Java, Kotlin,
C#, Ruby, PHP and Swift, with skips that depend on the environment left alone (testing.Short(),
@DisabledOnOs, skip ... unless, XCTSkipUnless, #[cfg_attr(..., ignore)]). 29 labeled cases
were added: 64 of 64 cheats caught and 44 of 44 look-alikes silent.

Pipelines and project.

• The Action takes comment: true to keep one pull request comment up to date, and stays quiet
  on a clean pull request.
• CI runs the full suite on macOS and a first-run smoke test on Linux, macOS and Windows. A release
  workflow can publish with npm provenance once it is enabled (docs/maintainers.md).
• README rewritten as a landing page; the long reference moved to docs/ (rules, integrations,
  reference, comparison). Added CONTRIBUTING, SECURITY, a code of conduct, pull request and issue
  templates for false positives and missed shortcuts, and dependabot. Internal plans and evidence
  moved to docs/archive.

Known limits: the agent hooks follow each agent's documented protocol and are tested with simulated
hook input, not yet in long live sessions. assertion-weakened is still JavaScript, TypeScript and
Python only.

Install: npx agent-roast demo · npx agent-roast hook install · Action Nari-Ab/agent-roast@v1 · Claude Code: /plugin install agent-roast --marketplace Nari-Ab/agent-roast

Full changelog: https://github.com/Nari-Ab/agent-roast/blob/main/CHANGELOG.md

v0.11.0 — Replay your recent commits

agent-roast 0.11.0 adds a command that replays your recent commits one at a time, so the first run shows what a pull-request check would have said, and removes a class of false alarms from test-skip.

npx agent-roast@0.11.0 replay

• agent-roast replay: reviews each of the newest 50 commits (--last N) on its own against its parent, following first-parent history, and lists the commits that would have been blocked or drew a warning about a weakened check, with the evidence. It never fails because of a finding, and a commit too large to read is reported instead of stopping. Line patterns such as as any are counted and listed with --all-rules; --json returns every finding. On 179 public repositories that use coding agents (the newest 50 commits of each, 7,905 commits), no commit would have been blocked in 149 of them; 0.6% of commits would have been blocked and 12.1% drew a warning.
• --advisory for review, check and --staged: report without failing, the same as an empty --fail-on. cmd.exe and Windows PowerShell 5.1 do not pass the empty argument through, so the documented advisory command failed there.
• Fewer false alarms in test-skip, which blocks by default. Skips that depend on the environment are no longer reported: Playwright's test.skip(condition, 'reason'), pytest.skip or self.skipTest under if / try, and pytest.skip in a fixture or helper. Skips that name a test, skip(true), a skip that is a direct statement of a test function, and conditions that mention CI are still reported. On the 75 commits of the corpus that carried a test-skip finding, findings fell from 160 to 34. The cost: a conditional skip that hides a failing test is not reported unless its condition mentions CI.
• A review ref that does not resolve now says why: a shallow clone, or a history shorter than the requested HEAD~N (with the number that works).
• docs/pilot.md starts with replay, and the Action description no longer presents the tool as an audit of AI shortcuts and panic loops.

Method and numbers: docs/benchmarks.md, section 4.

The v1 Action tag installs the CLI version pinned in action.yml; use Nari-Ab/agent-roast@v0.11.0 to pin this release exactly.

v0.10.1 — Blocking findings shown first

agent-roast 0.10.1 fixes a report that could hide a blocking finding, and adds a guide for trying the tool on your own repository.

npx agent-roast@0.10.1 review --base origin/main

• Fix: the terminal report shows at most 20 findings, in file order. A change with 20 or more warnings in files that sort earlier (typically as any in a large pull request) left a finding such as assertion-weakened behind "N more new findings", with only the BLOCKED line naming the rule. Advisory runs (--fail-on '') had the same problem. Findings are now ordered by importance in the terminal, markdown and GitHub summary output: rules that block, then the before/after rules and the default blocking shortcuts, then the noisier line patterns. JSON and SARIF output are unchanged.
• Docs: docs/pilot.md shows how to try the tool on your own repository (a five-minute look back, an advisory workflow that never fails a pull request, a feedback form).

The v1 Action tag installs the CLI version pinned in action.yml; use Nari-Ab/agent-roast@v0.10.1 to pin this release exactly.

v0.10.0 — Measured on real commits, quieter rules

agent-roast 0.10.0 is the release where the tool was measured on real history and then made quieter.

What was measured: 12,645 commits that touch tests or CI/config, from 166 public repositories (AI-signed and other commits side by side). Weakened checks are rare (about 1 in 300 such commits triggers a block), usually deliberate and explained, and not more frequent in AI-signed commits (odds ratio 0.81, 95% CI 0.40-1.67 inside the same repository). About one finding in five was a false alarm; this release removes the classes of false alarm that reading the findings revealed.

npx agent-roast@0.10.0 review --base origin/main
npx agent-roast@0.10.0 --staged

• Behavior change: assertion-weakened now means an exact check replaced by a weaker one and
  still blocks. A test that only lost assertions is the new assertion-removed rule, which warns:
  most such cases were deliberate or false alarms. A policy that set "assertion-weakened": "off"
  to silence lost assertions should set "assertion-removed": "off" instead.
• guard-weakened now reports a workflow that keeps its tests but stops running on pushes and pull
  requests, and a deleted workflow that gated changes. It no longer reports: tolerated failure on a
  step or job added in the same change; a lint rule enforced by a namespaced twin or switched off
  only in an override; CI restructuring that keeps test steps (multi-line run blocks, shared
  workflows, tests moved to another workflow); or scheduled-only workflows.
• Fewer false alarms in the assertion rules: formatter rewraps and trailing remarks are ignored;
  Python assert_* helpers (mocks, numeric comparisons) count as checks; checks moved into an
  assertion helper introduced by the same change, checks whose literals are still asserted in the
  test, Testing Library getBy* presence checks, and not-null checks flipped to presence checks are
  not reported.
• Add scripts/benchmark-agents.mjs (AI-signed versus other commits, Wilson intervals,
  per-repository comparison), a --cli option for benchmarking a frozen install,
  benchmarks/agent-corpus.tsv (the 166 repositories and their revisions), and 76 labeled cases
  (40 cheats, 33 harmless changes, 3 documented gaps). Method and results: docs/benchmarks.md.
• The README no longer presents the tool as an AI-cheating detector: the measurements show these moves are not specific to AI.
• The SARIF rule list and schema.json include assertion-removed.

Method, corpus, hand labels and limits: docs/benchmarks.md. The labels are one reader's opinion, and the numbers describe committed history, not what an agent tries during a session.

Full changelog: https://github.com/Nari-Ab/agent-roast/blob/main/CHANGELOG.md

v0.9.0 — Catch weakened tests and quality gates

agent-roast now catches the other half of "make the checks pass": changes that weaken the tests and quality gates instead of fixing the code.

npx agent-roast@0.9.0 review --base origin/main
npx agent-roast@0.9.0 --staged

• New diff-aware rules for review and --staged that catch changes which make a project
  easier to pass without making it more correct:
  • assertion-weakened: a test loses assertions, or an exact check becomes a weaker one.
  • test-removed: a test, or a whole test file, disappears (renames and moves are not reported).
  • expectation-rewritten: an expected value changes while the inputs stay the same and
    production code changed in the same review.
  • guard-weakened: lower coverage thresholds, compiler strictness switched off, lint rules
    downgraded, CI steps allowed to fail or no longer running tests, test scripts that cannot fail.
• Behavior change: assertion-weakened and guard-weakened block by default, like
  test-skip. To relax, set "rules": { "assertion-weakened": "warn" } (or "off") in the
  trusted policy. test-removed and expectation-rewritten report without blocking.
• Comparison is statement-based and formatter-safe: quote style, wrapping and trailing commas are
  ignored; moved, renamed, parametrized and snapshot-folded tests are not reported. Parsing is
  bounded by the next test's start, so work stays linear on very large or malformed test files.
• Add benchmarks/cases.mjs (32 cheats, 15 controls, 3 documented gaps),
  scripts/benchmark-recall.mjs and scripts/benchmark-noise.mjs, and document the method and
  results in docs/benchmarks.md. On 606 test/config commits of five open-source projects the
  blocking rules fired on 7 (1.2%); each was read by hand.
• The SARIF rule list and schema.json include the new rules.
• Remove unreferenced launch assets (audio/video, GIF render scripts, Product Hunt kit, the old
  site video page) and the gifenc and @napi-rs/canvas dev dependencies.
• Update the composite Action to install exactly 0.9.0, and the site and README to match.

Measured on real history: the blocking rules fired on 7 of 606 test/config commits (1.2%) across commander.js, execa, click, requests and zod; each was read by hand (one real defect, six deliberate changes). Method, revisions and limits: docs/benchmarks.md. Accuracy on real AI-agent pull requests is not measured yet.

Full changelog: https://github.com/Nari-Ab/agent-roast/blob/main/CHANGELOG.md

v0.8.1 — Fixes from real CLI testing

Agent Roast 0.8.1 fixes issues found by testing the published CLI in disposable Git projects. No new runtime dependencies.

• Running init inside a package now creates policy and CI at the repository root, preserves existing files, and reports the actual destination. Bare repositories are refused before writes.
• Regexp examples after JS arrow/throw/control headers no longer create focused-test false alarms; contiguous source context is preserved.
• Multiline empty catches, TypeScript bypass casts and placeholder errors remain detectable after formatting. Deleting the only handler from a formatted catch is caught during review. Missing diff hunks are never joined.
• Repeated unfinished catch headers are bounded, and ordinary TODO domain errors produce less noise.
• Added 39 regression tests and a reusable 22-scenario CLI workflow check, now included in CI on Node 18/20/22.

Validation: 486 tests pass locally; the installed release archive passes all 22 user workflows on Node 18.20.8, 20.19.5 and 22.20.0. GitHub CI runs the same suite and workflows.

Remaining scope: Node test option objects ({skip: true} / {only: true}), template interpolation, custom aliases and complex mixed-language syntax are not fully analyzed. These are curated engineering checks, not a production accuracy measurement or a substitute for application tests.

v0.8.0 — Trusted exceptions and safer code checks

Agent Roast 0.8.0 makes review decisions explicit and closes gaps in pre-commit checks. Accept one intentional finding with a reason and expiry while additional identical occurrences still block; context reviews continue using trusted base/HEAD policy.

• Added precise fingerprint/count/reason/expiry exceptions, --ignore-exceptions, visible accepted records and SARIF suppressions. Baseline and exception allowances cannot stack to hide new duplicates. Included an offline editor configuration schema.
• Fixed actual git commit -a / --only candidate indexes. Ambient Git variables cannot redirect an explicitly selected project. Bounded reads, safe initialization and strict policy validation reject links, special files, invalid settings and incomplete scans.
• Fixed language-specific false positives and missed multiline/modifier test registrations, placeholder macros and assertions. Hardened history record framing, ignore-glob matching and source parsing against malformed or costly input.
• Escaped untrusted terminal/Markdown/SVG data and stabilized SARIF occurrence identities. Updated site instructions and added explicit Action advisory: true support; incomplete scans still fail.
• Updated development-only Vitest/Vite/esbuild to patched versions. No new runtime dependencies.

npx agent-roast@0.8.0 check --format json
npx agent-roast@0.8.0 --staged
npx agent-roast@0.8.0 review --base origin/main

Compatibility: the CLI supports Node.js 18+. Building/testing requires Node.js 20.19+ or 22.12+. Unknown configuration keys and non-string severity settings now fail explicitly. Historical/legacy scans return exit 2 when supported source was skipped instead of claiming a clean pass. Context review still gates introduced findings; use a full check to revisit old findings after exception expiry.

Validation: 447 tests, CLI/real-hook checks on Node18/20/22 with a supported test runner, TypeScript checking, tarball installation and JSON Schema/SARIF validation. npm audit reported zero known runtime or development vulnerabilities at release preparation. Fourteen deliberately selected reproductions of known 0.7.0 defects now match expected outcomes; this is regression coverage, not production accuracy, customer adoption or ROI evidence.

The analyzer remains a focused heuristic review aid alongside compilers, tests and language-aware linters. Custom aliases, computed members and template interpolation are not fully parsed. No live GitHub SARIF upload was performed. The composite Action installs exactly 0.8.0; v1 is moved after public npm availability is verified.

v0.7.0 — Safer pre-commit checks and focused-test detection

Agent Roast 0.7.0 makes pre-commit checks compare complete HEAD/index source, so deleting the only catch handler is detected and examples inside existing comments no longer trigger findings. Existing debt stays separate from new blocking findings.

• Added test-focus for .only, fit, fdescribe and common JS/TS test modifier chains, with guidance, severity configuration, baseline support and SARIF output.
• Added review --staged; the existing --staged CLI uses the same context comparison. Partial staging, first commits, renames and linked worktrees are supported. Unmerged entries, incomplete source, cancellation and concurrent HEAD/index changes cannot produce a successful result.
• Staged policy comes from HEAD; first commits use built-in defaults. Candidate config edits cannot silently weaken the check. Added .mts and .cts source support and fixed copyable configuration examples.

npx agent-roast@0.7.0 --staged
npx agent-roast@0.7.0 review --base origin/main

Migration: staged CLI JSON now uses ReviewResult (mode: review, source: index, schema 1.1.0). headSha is an INDEX: snapshot digest, and EMPTY denotes an unborn base. The programmatic auditPullRequest({staged:true}) retains its old added-lines contract; use reviewStagedChanges() for context comparison. Existing explicit failOn lists must include test-focus to make it block.

Validation: 275 tests on Node.js 18/20/22, TypeScript typecheck, npm tarball installation and staged SARIF smoke tests. Nine curated synthetic staged changes improved from 4/9 expected outcomes in 0.6.0 to 9/9; this is regression coverage, not a production accuracy or adoption claim. No new runtime dependencies.

The composite Action installs exactly 0.7.0; the v1 tag is updated after npm availability is verified.

v0.6.0 — Context-aware PR checks and SARIF

Agent Roast 0.6.0 reviews full before/after source in a pull request so it can detect shortcuts introduced by deleting code, while keeping existing findings separate from new ones.

Try it

npx agent-roast@0.6.0 review --base origin/main --head HEAD
npx agent-roast@0.6.0 review --base origin/main --format markdown
npx agent-roast@0.6.0 check --format sarif > agentroast.sarif

What changed

• New review workflow compares complete source blobs at merge-base and head. It catches cases such as deleting the only error handler inside a catch block, without requiring a manually generated baseline.
• Reports introduced, existing and no-longer-detected patterns, preserving occurrence counts and Git-recognized renames. A disappeared pattern is not claimed to be a verified behavioral fix.
• Policy comes from the explicitly supplied base commit. Candidate policy changes are reported and ignored for the current review.
• Offline SARIF 2.1.0 output is available for check and review, with rule guidance, relative locations, occurrence fingerprints and incomplete-run status.
• Review reads immutable Git objects without checking out source or running project code. Reads are bounded and cancellable; incomplete comparisons exit 2.
• GitHub Action defaults to mode: review, with mode: audit for legacy added-line checks. Installation runs outside candidate npm config. pull_request_target is rejected.

Validation and limits

The release includes 212 tests and a transparent, project-maintained corpus of 14 synthetic changes. Context review matches the expected finding sets in 14/14 cases, compared with 10/14 for the added-line workflow. These are regression controls, not a production accuracy or customer-adoption claim.

Detectors remain targeted heuristics, not an AST/type analyzer or a replacement for tests, compilers and security tools. Comparison covers changed supported source files. Limits are 1,000 changed files, 2 MiB per source blob and 50 MiB aggregate source. Partial clones and ambiguous multiple merge-bases are unsupported; use complete history (fetch-depth: 0). Some deletion-only findings may appear in summaries rather than inline GitHub annotations.

Action compatibility

The new default reviews full source context and uses base-branch policy. Set mode: audit to retain the previous added-line workflow. The Action installs exactly agent-roast@0.6.0.