v1.0.1

What's New

• The HTML report from unsafe-eval report --format html follows the leaderboard's current design. The data and numbers shown are unchanged. (#16)

Bug Fixes

• When a gateway such as OpenRouter relays an upstream failure as a choice with finish_reason: "error", the sample is generated again, up to three times, and then fails instead of being scored as an invalid output. A relayed content block is still a refusal. (#15)
• The CLI banner leaves a blank line above itself. (c7d70fe)

Internal

• Raise the minimum SDK versions of the extras: google-genai>=2.29.0,<3, openai>=3.28.0, anthropic>=1.13.0, typesafe-sdk>=0.7.4. (#6, #8)
• Docstring updates, a README badge fix, and dependency bumps. (#3–#14, #15, #17, #19–#22)

v1.0.0

First public release of unsafe-eval, a framework for reproducible prompt-safety evaluation built on Inspect AI. It scores safety classifiers and general-purpose models on the same benchmarks with the same rules, so their results can be compared directly.

🚀 What's New

• Add adapters for 8 prompt-safety benchmarks, each loaded at a pinned dataset revision: SimpleSafetyTests, HarmBench Text Test, Aegis 2.0, OpenAI Moderation, ToxicChat 0124, WildGuardMix, CoSApien and DynaGuardrail.
• Add 15 versioned protocols, one per model family, that fix the prompt format, generation settings, allowed options, scorer and metrics: shieldstral-1.0-3b, scope-guard-2601, scope-guard-2609, nemotron-3.5-content-safety, polyguard-qwen, qwen3-guard-gen, gpt, claude, gemini, jev, d1, clef, laya, gliner2.5-decide and gliguard.
• Ship 502 packaged run configurations covering 64 models, which pin the dataset revision and, where the provider allows it, the model revision (an exact hub commit for local models, a dated snapshot for hosted ones).
• Add Inspect model providers for models that no stock provider reaches: TypeSafe's Jev, Kev, Laya, Clef, d1, CLM, GLiNER2 and Orbitals' ScopeGuard. Each provider and runtime is an optional extra (vllm, hf, openai, openrouter, anthropic, google, typesafe, orbitals, laya, clef, clm, gliner).
• Add the unsafe-eval CLI: run executes packaged or local configurations with --resume, list shows benchmarks, protocols and configurations, config path locates a packaged configuration, and report summarizes the metrics files of a log directory.
• Record the setup of every run (unsafe-eval and inspect-ai versions, uv.lock hash, git commit and dirty state) in its metrics file, and ask before mixing runs from different setups in one log directory or reusing them with --resume.
• Write an HTML report (report.html) to the log directory after every run, or on demand with report --format html: a single file that needs no network, styled like the public leaderboard, with per-benchmark metrics and median latency.

🔧 Internal

• Derive the package version from git tags with hatch-vcs.
• Publish to PyPI through trusted publishing when a GitHub release is published (.github/workflows/publish.yaml).
• Document every benchmark, protocol, metric and provider under docs/, with guides for adding a benchmark, a model and a metric.