First tagged release of LLMSecTest — an open-source, pytest-based security test suite for LLM applications, covering the OWASP LLM Top 10 (2025) with CI/CD-ready reports.
Pre-1.0 and built in the open. This is the first tagged GitHub release; it is not yet published to PyPI. The forward-looking plan is the roadmap.
Added
- CVSS v4.0 scoring. Each OWASP category carries a representative
CVSS:4.0base vector; findings report its base score as the SARIFsecurity-severity. Ten canonical scores ship dependency-free; the optionalcvssextra scores custom vectors. - Every scan covers all ten OWASP categories — no silent gaps. Implemented categories run real probes; not-yet-implemented ones are reported as skipped tests marked not yet implemented, and every run prints a coverage footer.
- Black-box testing of a real application.
--target app:<url>drives your running app through its own HTTP endpoint; a persona proxy tests an app's real system prompt against a local model. Covers LLM01/LLM05 out of the box, plus LLM02/LLM06/LLM07 with the relevant inputs. - Local and self-hosted models. Ollama adapter and OpenAI-compatible
base_url— run with no API key and no paid calls. - OWASP probe suite. Adapter-driven probes for LLM01, LLM02, LLM05, LLM06 and LLM07 with false-positive-resistant detectors.
- Reporting. SARIF v2.1.0, HTML, JSON and Markdown reports with OWASP metadata, CWE tags, compliance mapping, risk scoring, baselines and policy gates.
- Unified LLM adapter. OpenAI, Anthropic and Hugging Face behind one interface, plus offline test doubles.
- Command-line interface. The
llmsectestconsole script. - Documentation site at https://docs.llmsec.dev.
Changed
- Reconciled all OWASP metadata to the OWASP LLM Top 10 (2025) list.
- SARIF
security-severitynow carries the real CVSS v4.0 base score.
Fixed
- CLI: a space-separated option value (e.g.
--report-dir tmp) was mistaken for a positional test path and silently skipped the packaged suite.
Full changelog: https://github.com/wehnsdaefflae/llmsectest/blob/main/CHANGELOG.md