Skip to content

v0.1.2 - public benchmark

Choose a tag to compare

View all tags
@Sonofg0tham Sonofg0tham released this 16 Jun 15:14
v0.1.2
72a5e5b

Ward now has a measured detection envelope

The honest-assessment workflow that drove v0.1.1 said the single
highest-leverage move was "ship a public benchmark". This release does
it.

`ward bench`

```bash
ward bench

Wrote benchmark report: ward-bench-report.md

In-scope recall: 75.2% FPR: 0.0%

```

Scores Ward against four bundled public corpora, each redistributable
under their upstream MIT or Apache 2.0 licence:

Corpus Fit Result
lakera_ignore_instructions in-scope 68.0% recall
deepset_prompt_injections in-scope 40.0% recall, 0.0% FPR
spikee_jailbreaks in-scope 100.0% recall
advbench_harmful_behaviors ceiling test 0.0% (by design)

In-scope total: 75.2% recall, 0.0% false-positive rate on bundled
50-row samples per corpus.

Full report bundled in `benchmark/v0.1.2.md`.

What this gets us

  • A measured detection envelope instead of a marketing claim.
    Every Ward release now commits its own benchmark numbers so the
    history is auditable.
  • Honest ceiling reporting. AdvBench is included as a deliberate
    ceiling test - bare harmful-intent strings with no injection
    phrasing. Ward will always score 0% there, and that's the right
    framing.
  • Closed gap from the inaugural numbers. Inaugural was 60.0%
    recall. Rule expansion driven by the Lakera misses raised it to
    75.2% in the same release. `io.ignore_previous` now catches
    diversified phrasings ("ignore all previous text", "ignore the
    directions above", "ignore all your instructions"). New
    `io.reveal_instructions` rule catches the prompt-extraction class.

Honest caveats

Remaining gaps documented in `SECURITY.md`: typo
tolerance, role-play injections, single-space character spacing,
GCG / AutoDAN / Crescendo, ASCII art, Unicode TAG block, multimodal,
RAG. Those are v0.2 LLM-judge tier territory, not regex territory.

By the numbers

  • 190 tests (up from 179)
  • 4 bundled adversarial corpora, ~200 sample rows
  • New `ward bench` CLI command + JSON / Markdown reporters
  • mypy strict, ruff clean, format clean
  • CI green on Ubuntu / macOS / Windows x Python 3.11 / 3.12 / 3.13

Upgrade

```bash
pipx upgrade ward-scanner
```

Assets 2
Loading