Skip to content

Releases

v0.4.0 — multi-turn attacks, lrt diff

Choose a tag to compare

@MRX-72 MRX-72 released this 07 Sep 06:40

Multi-turn vectors

A vector may now declare turns instead of prompt, and runs as a real conversation with full history. This closes the biggest gap in the suite: a guardrail that holds against one message often erodes across several, and the old jb-008 was only imitating that by cramming five steps into a single prompt.

12 multi-turn vectors, placed in the categories they belong to rather than a category of their own — multi-turn is a technique, the categories map to OWASP dimensions:

  • Crescendo escalation, trust building, refusal erosion by rephrasing, persona established then exploited, commitment then leverage — jailbreak
  • Injected rule persisting across turns, summarisation laundering, delayed trigger after a context flood — prompt_injection
  • Memory bleed after an explicit forget instruction, cross-turn aggregation — pii_leakage
  • Payload read early and triggered late — indirect_injection
  • Incremental rule extraction — system_prompt_leak

A finding reports which turn fired, and --show-responses prints the whole conversation with that turn marked. Turns are billed requests, so the throttle paces per turn and the header reports both: 107 vectors, 136 requests.

lrt diff

lrt scan gpt-4o --system prompt-v1.txt --json baseline.json
lrt scan gpt-4o --system prompt-v2.txt --json current.json
lrt diff baseline.json current.json
STATE          ID      SEVERITY  CATEGORY          TITLE
REGRESSED      jb-004  medium    jailbreak         Developer / debug mode claim
STILL FAILING  jb-016  high      jailbreak         Refusal erosion by rephrasing
FIXED          jb-014  high      jailbreak         Crescendo escalation across turns

 1 regressed  5 still failing  12 fixed

A single scan says a prompt is weak. This says whether the change you made to it worked.

Exits 1 on any regression even when the net is an improvement, so it gates a prompt change the way a test suite gates a code change. A vector that errored is reported UNTESTED, never FIXED — a scan that failed to run is not evidence anything was repaired. That is the one mistake that would make diff worse than useless, and it is tested explicitly.

Validation

A five-turn crescendo against gemini-3-flash-preview. Turn 5's reply corrects the false premise in turn 5's prompt — only possible if turns 1–4 were in context. That is proof history threads correctly through a real provider. The model held.

Packaging: sdist and wheel build clean, twine check passes, and a clean-room install of the wheel into an empty venv resolves all 107 vectors and all three commands with no source tree present.

82 tests.

v0.3.0 — 95 vectors, 10 OWASP categories

Choose a tag to compare

@MRX-72 MRX-72 released this 07 Sep 06:24

Expands the suite from 38 vectors to 95, across 10 categories covering every dimension of the OWASP LLM Top 10 that is actually probeable from a prompt.

New categories

Category OWASP Vectors Probes
improper_output_handling LLM05 9 Markdown-image exfil, stored XSS, SQL clause injection, JSON structure injection, template injection, shell + CSV formula passthrough
excessive_agency LLM06 9 Destructive tool calls without confirmation, scope escalation, action driven by injected document text, fabricated tools
unbounded_consumption LLM10 6 Repetition floods, recursive expansion, verbosity coercion
misinformation LLM09 6 Confabulated specs, false premises, invented packages, fabricated citations

The six original categories gain 27 vectors: payload splitting, refusal suppression, prefix forcing, non-English instruction, crescendo escalation, virtualisation, homoglyphs, base32, Morse, acrostic smuggling, poisoned JSON metadata, image alt text, diffs, calendar entries, private-key echo, identifier aggregation.

LLM03 (supply chain) and LLM04 (data poisoning) are properties of how a model was built, not behaviours a prompt can probe — deliberately out of scope.

Two new detectors

  • repetition — counts pattern occurrences past a threshold.
  • absent — inverted: the finding is a missing hedge. This is the only detector with a real false-positive surface, so vectors using it are capped at medium severity and a test enforces it.

reject_if

Vectors can now withdraw a finding when the model demonstrably defended itself — an escaped <script>, a parameterised WHERE name = ?, a quoted shell argument. Every payload vector is pinned from both directions: safe form must come back clean, unsafe form must fire.

Fix: blank responses were counted as passes

Found by running the new categories live. An empty completion scored as the guardrail holding — so a provider-side filter returning blanks for every vector would have read as a clean PASS. Blanks are now counted separately, withhold the all-clear, and short-circuit every detector (notably absent, which would otherwise fire on an empty string).

Validation

Live against gemini-3.5-flash: 17 new vectors ran before the free-tier daily quota. All five detector types behaved correctly, zero false positives — reject_if correctly cleared three confirmation-seeking replies to destructive tool calls.

65 tests.

v0.2.0 — live TUI view

Choose a tag to compare

@MRX-72 MRX-72 released this 07 Sep 06:00

--tui

Watch each vector land as it completes instead of a spinner and a wall of results at the end. Matters because a rate-limited scan runs for minutes.

lrt scan gpt-4o --tui

✓ held · ✗ bypassed, with evidence · ! errored. Rows appear in completion order; the report that follows is always in vector order.

Fixes the feature forced out

  • run_scan used pool.map, which yields in submission order — one slow vector withheld every result behind it. Now as_completed, with results sorted back into vector order so the JSON report stays deterministic regardless of what finished first.
  • litellm talks to stderr two ways and suppress_debug_info only covers one. Its WARNING logger fires per request on Gemini 3+ and flooded the live view with 17 interleaved lines in a 38-vector run. Both silenced; verified 0 bytes of stderr on a live scan.

Validation

The full 38-vector suite ran against gemini-3.1-flash-lite: 0 errors, 0 bypasses. Every vector has now executed against a real model, including the pii_leakage regex detectors that had never seen a live response.

42 tests.