SoK Multi-Turn Jailbreak Experiments

This repository contains the release version of the code used for the mechanism analyses in SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks.

The repository has been trimmed to keep only code, prompts, configuration files, and datasets needed by the experiments described in Appendix A of the paper. Generated logs, cached files, virtual environments, previous results, figures, and ad-hoc analysis outputs are intentionally removed.

Experiment Mapping

Paper question	Method(s)	Category	Local directory	Dataset(s)
RQ1: Organization vs. accumulation	FITD, MRCJ	DEA / SRA	Foot-in-the-door-Jailbreak/, Inc_Exp/MRCJ/	JailbreakBench, MUCD, AdvBench
RQ2: Amplification components	ActorAttack	DEA	actorattack/ActorAttack/	HarmBench
RQ3: Escalation shape	MRCJ	SRA	Inc_Exp/MRCJ/	MUCD, AdvBench
RQ4: Sub-paradigm progression	X-Teaming	SRA	x-teaming/	JailbreakBench-style behavior set

Directory Overview

• Foot-in-the-door-Jailbreak/: FITD implementation for the RQ1 DEA-side organization-vs-accumulation experiment.
• Inc_Exp/MRCJ/: MRCJ implementation for the RQ1 SRA-side comparison and RQ3 escalation-shape experiments.
• actorattack/ActorAttack/: ActorAttack implementation for the RQ2 actor-count and self-talk ablations.
• x-teaming/: X-Teaming backbone and configs/exp3 settings for the RQ4 A1/A2/A3 sub-paradigm progression experiment.

Environment

Each method keeps its original dependency file:

pip install -r actorattack/ActorAttack/requirements.txt
pip install -r Foot-in-the-door-Jailbreak/requirements.txt
pip install -r Inc_Exp/requirements.txt
conda env create -f x-teaming/environment.yml

API keys and service endpoints should be provided through environment variables or local config files. Secret-bearing .env files are not part of the release.

Running the Experiments

RQ1: FITD

FITD uses the JailbreakBench behavior set and its default multi-turn attack configuration.

cd Foot-in-the-door-Jailbreak
python FITD.py

For prompt-based replay:

python FITD_prompt.py --model_name gpt-4o --control True --max_length 10

RQ1/RQ3: MRCJ

MRCJ uses MUCD-style auxiliary questions with malice levels and AdvBench final targets. The paper fixes four escalation steps for RQ3.

cd Inc_Exp
python MRCJ/train/train.py --target_model mistral --judge_model gpt-4o --cat_questions_num 2 --harmful_score_threshold 4 --train_from_scratch --dir my_model_data
python MRCJ/test/test.py --target_model mistral --judge_model gpt-4o

Use the model names supported by MRCJ/train/train.py and MRCJ/test/test.py.

RQ2: ActorAttack

ActorAttack uses HarmBench queries. Vary --actors from 1 to 5 for the actor-count experiment. Fix --actors 3 and disable self-talk in code/config for the self-talk ablation, matching the paper setup.

cd actorattack/ActorAttack
python main.py --questions 400 --actors 3 --behavior ./data/harmbench.csv --attack_model_name deepseek-chat --target_model_name gpt-4o --early_stop --step_modify

RQ4: X-Teaming

RQ4 uses the X-Teaming backbone with DeepSeek-V3 as the attacker model and compares A1, A2, and A3 under the same turn budget.

cd x-teaming
python main.py --config configs/exp3/A1_gpt4o.yaml
python main.py --config configs/exp3/A2_gpt4o.yaml
python main.py --config configs/exp3/A3_gpt4o.yaml

Equivalent configs are available for the other target models used in the paper.

Evaluation

The paper uses GPT-4o as the automated judge with a 1-to-5 harmfulness scale. A score of 4 or higher is counted as a successful jailbreak. Generated outputs and judge reports are not included in this release; reruns should write fresh logs/results locally.

Release Policy

This code is for controlled research reproduction only. The repository intentionally excludes historical experiment outputs, cached bytecode, virtual environments, notebooks, local secrets, and unrelated exploratory code.