Skip to content

v0.4.0 — multi-turn attacks, lrt diff

Latest

Choose a tag to compare

@MRX-72 MRX-72 released this 07 Sep 06:40
· 90 commits to main since this release

Multi-turn vectors

A vector may now declare turns instead of prompt, and runs as a real conversation with full history. This closes the biggest gap in the suite: a guardrail that holds against one message often erodes across several, and the old jb-008 was only imitating that by cramming five steps into a single prompt.

12 multi-turn vectors, placed in the categories they belong to rather than a category of their own — multi-turn is a technique, the categories map to OWASP dimensions:

  • Crescendo escalation, trust building, refusal erosion by rephrasing, persona established then exploited, commitment then leverage — jailbreak
  • Injected rule persisting across turns, summarisation laundering, delayed trigger after a context flood — prompt_injection
  • Memory bleed after an explicit forget instruction, cross-turn aggregation — pii_leakage
  • Payload read early and triggered late — indirect_injection
  • Incremental rule extraction — system_prompt_leak

A finding reports which turn fired, and --show-responses prints the whole conversation with that turn marked. Turns are billed requests, so the throttle paces per turn and the header reports both: 107 vectors, 136 requests.

lrt diff

lrt scan gpt-4o --system prompt-v1.txt --json baseline.json
lrt scan gpt-4o --system prompt-v2.txt --json current.json
lrt diff baseline.json current.json
STATE          ID      SEVERITY  CATEGORY          TITLE
REGRESSED      jb-004  medium    jailbreak         Developer / debug mode claim
STILL FAILING  jb-016  high      jailbreak         Refusal erosion by rephrasing
FIXED          jb-014  high      jailbreak         Crescendo escalation across turns

 1 regressed  5 still failing  12 fixed

A single scan says a prompt is weak. This says whether the change you made to it worked.

Exits 1 on any regression even when the net is an improvement, so it gates a prompt change the way a test suite gates a code change. A vector that errored is reported UNTESTED, never FIXED — a scan that failed to run is not evidence anything was repaired. That is the one mistake that would make diff worse than useless, and it is tested explicitly.

Validation

A five-turn crescendo against gemini-3-flash-preview. Turn 5's reply corrects the false premise in turn 5's prompt — only possible if turns 1–4 were in context. That is proof history threads correctly through a real provider. The model held.

Packaging: sdist and wheel build clean, twine check passes, and a clean-room install of the wheel into an empty venv resolves all 107 vectors and all three commands with no source tree present.

82 tests.