Skip to content

v0.2.1 - real reviewer-agent lab

Choose a tag to compare

@Sonofg0tham Sonofg0tham released this 02 Jul 16:48

Put a real AI reviewer behind Ward and see the difference

ward lab attack proved whether an injection reaches an agent's
context. ward lab review (new) goes further: it puts a real reviewer
agent behind Ward and shows what the agent does - the before/after
that is the whole portfolio thesis.

# Offline reviewer (no API key) - deterministic, models a naive agent:
ward lab review

# Real Claude reviewer:
pip install "ward-scanner[judge]" && export ANTHROPIC_API_KEY=sk-...
ward lab review --reviewer anthropic

The asymmetry, on the six bundled malicious PRs (offline)

Count
Blocked by Ward before the reviewer ran 6 / 6
Naive reviewer approved the malicious PR without Ward 5 / 6
Naive reviewer approved the malicious PR with Ward 0 / 6

Framed honestly

The report never claims "the model always gets hijacked" - modern models
often resist. The point it makes is the honest one: with Ward the
injection is refused before the reviewer's context is populated
(deterministic, provable, model-agnostic); without Ward, whether the
model resists is a gamble.
Ward turns "hope the model resists" into
"the model never sees it".

The AnthropicReviewer uses a realistic, deliberately un-hardened
prompt - a typical developer-built reviewer, i.e. the thing being
protected, in deliberate contrast to the hardened judge tier.

Also in this release

  • RELEASING.md - the maintainer runbook for PyPI (trusted-publisher
    OIDC setup) and GitHub Marketplace, so pipx install ward-scanner
    becomes real.

By the numbers

  • 250 tests (up from 233)
  • New src/ward/lab_reviewer.py + ward lab review command
  • Reuses the judge tier's Claude plumbing (injectable client, lazy import,
    graceful degradation with no key)
  • Detection benchmark unchanged (75.2% smoke / 53.5% full, 0.0% FPR) -
    this is portfolio/demo tooling, not a detection change
  • mypy strict, ruff clean, CI green across the matrix

Upgrade

pipx install "ward-scanner[judge]"   # reviewer lab with real Claude
- uses: sonofg0tham/ward@v0.2.1