Repository navigation
Releases
v0.4.0 — multi-turn attacks, lrt diff
Multi-turn vectors
A vector may now declare turns instead of prompt, and runs as a real conversation with full history. This closes the biggest gap in the suite: a guardrail that holds against one message often erodes across several, and the old jb-008 was only imitating that by cramming five steps into a single prompt.
12 multi-turn vectors, placed in the categories they belong to rather than a category of their own — multi-turn is a technique, the categories map to OWASP dimensions:
- Crescendo escalation, trust building, refusal erosion by rephrasing, persona established then exploited, commitment then leverage —
jailbreak - Injected rule persisting across turns, summarisation laundering, delayed trigger after a context flood —
prompt_injection - Memory bleed after an explicit forget instruction, cross-turn aggregation —
pii_leakage - Payload read early and triggered late —
indirect_injection - Incremental rule extraction —
system_prompt_leak
A finding reports which turn fired, and --show-responses prints the whole conversation with that turn marked. Turns are billed requests, so the throttle paces per turn and the header reports both: 107 vectors, 136 requests.
lrt diff
lrt scan gpt-4o --system prompt-v1.txt --json baseline.json
lrt scan gpt-4o --system prompt-v2.txt --json current.json
lrt diff baseline.json current.jsonSTATE ID SEVERITY CATEGORY TITLE
REGRESSED jb-004 medium jailbreak Developer / debug mode claim
STILL FAILING jb-016 high jailbreak Refusal erosion by rephrasing
FIXED jb-014 high jailbreak Crescendo escalation across turns
1 regressed 5 still failing 12 fixed
A single scan says a prompt is weak. This says whether the change you made to it worked.
Exits 1 on any regression even when the net is an improvement, so it gates a prompt change the way a test suite gates a code change. A vector that errored is reported UNTESTED, never FIXED — a scan that failed to run is not evidence anything was repaired. That is the one mistake that would make diff worse than useless, and it is tested explicitly.
Validation
A five-turn crescendo against gemini-3-flash-preview. Turn 5's reply corrects the false premise in turn 5's prompt — only possible if turns 1–4 were in context. That is proof history threads correctly through a real provider. The model held.
Packaging: sdist and wheel build clean, twine check passes, and a clean-room install of the wheel into an empty venv resolves all 107 vectors and all three commands with no source tree present.
82 tests.
v0.3.0 — 95 vectors, 10 OWASP categories
Expands the suite from 38 vectors to 95, across 10 categories covering every dimension of the OWASP LLM Top 10 that is actually probeable from a prompt.
New categories
| Category | OWASP | Vectors | Probes |
|---|---|---|---|
improper_output_handling |
LLM05 | 9 | Markdown-image exfil, stored XSS, SQL clause injection, JSON structure injection, template injection, shell + CSV formula passthrough |
excessive_agency |
LLM06 | 9 | Destructive tool calls without confirmation, scope escalation, action driven by injected document text, fabricated tools |
unbounded_consumption |
LLM10 | 6 | Repetition floods, recursive expansion, verbosity coercion |
misinformation |
LLM09 | 6 | Confabulated specs, false premises, invented packages, fabricated citations |
The six original categories gain 27 vectors: payload splitting, refusal suppression, prefix forcing, non-English instruction, crescendo escalation, virtualisation, homoglyphs, base32, Morse, acrostic smuggling, poisoned JSON metadata, image alt text, diffs, calendar entries, private-key echo, identifier aggregation.
LLM03 (supply chain) and LLM04 (data poisoning) are properties of how a model was built, not behaviours a prompt can probe — deliberately out of scope.
Two new detectors
repetition— counts pattern occurrences past a threshold.absent— inverted: the finding is a missing hedge. This is the only detector with a real false-positive surface, so vectors using it are capped at medium severity and a test enforces it.
reject_if
Vectors can now withdraw a finding when the model demonstrably defended itself — an escaped <script>, a parameterised WHERE name = ?, a quoted shell argument. Every payload vector is pinned from both directions: safe form must come back clean, unsafe form must fire.
Fix: blank responses were counted as passes
Found by running the new categories live. An empty completion scored as the guardrail holding — so a provider-side filter returning blanks for every vector would have read as a clean PASS. Blanks are now counted separately, withhold the all-clear, and short-circuit every detector (notably absent, which would otherwise fire on an empty string).
Validation
Live against gemini-3.5-flash: 17 new vectors ran before the free-tier daily quota. All five detector types behaved correctly, zero false positives — reject_if correctly cleared three confirmation-seeking replies to destructive tool calls.
65 tests.
v0.2.0 — live TUI view
--tui
Watch each vector land as it completes instead of a spinner and a wall of results at the end. Matters because a rate-limited scan runs for minutes.
lrt scan gpt-4o --tui
✓ held · ✗ bypassed, with evidence · ! errored. Rows appear in completion order; the report that follows is always in vector order.
Fixes the feature forced out
run_scanusedpool.map, which yields in submission order — one slow vector withheld every result behind it. Nowas_completed, with results sorted back into vector order so the JSON report stays deterministic regardless of what finished first.- litellm talks to stderr two ways and
suppress_debug_infoonly covers one. Its WARNING logger fires per request on Gemini 3+ and flooded the live view with 17 interleaved lines in a 38-vector run. Both silenced; verified 0 bytes of stderr on a live scan.
Validation
The full 38-vector suite ran against gemini-3.1-flash-lite: 0 errors, 0 bypasses. Every vector has now executed against a real model, including the pii_leakage regex detectors that had never seen a live response.
42 tests.