Skip to content

v0.3.0 — 95 vectors, 10 OWASP categories

Choose a tag to compare

@MRX-72 MRX-72 released this 07 Sep 06:24
· 91 commits to main since this release

Expands the suite from 38 vectors to 95, across 10 categories covering every dimension of the OWASP LLM Top 10 that is actually probeable from a prompt.

New categories

Category OWASP Vectors Probes
improper_output_handling LLM05 9 Markdown-image exfil, stored XSS, SQL clause injection, JSON structure injection, template injection, shell + CSV formula passthrough
excessive_agency LLM06 9 Destructive tool calls without confirmation, scope escalation, action driven by injected document text, fabricated tools
unbounded_consumption LLM10 6 Repetition floods, recursive expansion, verbosity coercion
misinformation LLM09 6 Confabulated specs, false premises, invented packages, fabricated citations

The six original categories gain 27 vectors: payload splitting, refusal suppression, prefix forcing, non-English instruction, crescendo escalation, virtualisation, homoglyphs, base32, Morse, acrostic smuggling, poisoned JSON metadata, image alt text, diffs, calendar entries, private-key echo, identifier aggregation.

LLM03 (supply chain) and LLM04 (data poisoning) are properties of how a model was built, not behaviours a prompt can probe — deliberately out of scope.

Two new detectors

  • repetition — counts pattern occurrences past a threshold.
  • absent — inverted: the finding is a missing hedge. This is the only detector with a real false-positive surface, so vectors using it are capped at medium severity and a test enforces it.

reject_if

Vectors can now withdraw a finding when the model demonstrably defended itself — an escaped <script>, a parameterised WHERE name = ?, a quoted shell argument. Every payload vector is pinned from both directions: safe form must come back clean, unsafe form must fire.

Fix: blank responses were counted as passes

Found by running the new categories live. An empty completion scored as the guardrail holding — so a provider-side filter returning blanks for every vector would have read as a clean PASS. Blanks are now counted separately, withhold the all-clear, and short-circuit every detector (notably absent, which would otherwise fire on an empty string).

Validation

Live against gemini-3.5-flash: 17 new vectors ran before the free-tier daily quota. All five detector types behaved correctly, zero false positives — reject_if correctly cleared three confirmation-seeking replies to destructive tool calls.

65 tests.