v0.1.3 - real-world numbers, TAG-block, marketplace-ready
The credibility number
Ward now has a measured detection envelope against the full
1,391 real adversarial rows from four public corpora, not just the
smoke sample:
| Corpus | Rows | Recall | FPR |
|---|---|---|---|
| Lakera ignore_instructions | 777 | 57.8% | 0.0% |
| deepset prompt-injections | 546 | 21.7% | 0.0% (on 271 benign rows) |
| Spikee jailbreaks | 68 | 100.0% | 0.0% |
| AdvBench (ceiling test) | 520 | 0.0% (by design) | - |
In-scope total: 53.5% recall, 0.0% false-positive rate.
The 0.0% FPR on 271 benign rows is the strongest single number
in Ward's story. Full report: `benchmark/v0.1.3-full.md`.
Smoke number (bundled 50-row samples) stays at 75.2% for the CI
regression guard. The 21.7pp gap between smoke and full is the honest
signal about real-world adversarial diversification.
Reproduce it
```bash
pip install "ward-scanner[bench-download]"
ward bench --download lakera_ignore_instructions
--download deepset_prompt_injections
--download spikee_jailbreaks
--download advbench_harmful_behaviors
```
What else landed
Unicode TAG-block detection. Documented smuggling channel per
Embrace The Red and Riley Goodside. Any U+E0000-U+E007F char fires
`obf.unicode_tag` (severity HIGH) and the TAG-decoded text is
threaded through the rule engine so the smuggled instruction fires
its underlying rule too.
CI bench-as-guardrail. Every PR now runs `ward bench` on both
the base ref and the PR HEAD, diffs the numbers, and posts a sticky
comment. Silent regressions on the benchmark are impossible to merge
without a human noticing.
`ward bench-diff` CLI subcommand renders the delta between two
JSON reports as Markdown. Used by the CI workflow; also handy locally.
Full-corpus download mode. `ward bench --download `
caches to `~/.cache/ward/bench/` (Linux/macOS) or
`%LOCALAPPDATA%\ward\bench\ (Windows). Parquet corpora need the \[bench-download]` extra; CSV/JSONL work out of the box.
GitHub Marketplace ready. `action.yml` moved to the repo root
(marketplace requirement). Renamed to "Ward - Pre-Agent Metadata
Scanner". Users reference it as
`uses: sonofg0tham/ward@v0.1.3`.
By the numbers
- 201 tests (up from 190)
- 4 bundled corpora + optional full-corpus download
- New `ward bench-diff` CLI, new bench.yml workflow
- New `obf.unicode_tag` rule
- mypy strict, ruff clean, format clean
- CI green on Ubuntu / macOS / Windows x Python 3.11 / 3.12 / 3.13
Upgrade
```bash
pipx upgrade ward-scanner
```
Or in a workflow:
```yaml
- uses: sonofg0tham/ward@v0.1.3
with:
fail-on: high
```