Awesome MLLM/LLM Guardrails

Language: English | 中文

A curated map of recent LLM and MLLM safety guardrail resources: benchmarks, datasets, guard models, attack methods, evaluation suites, and deployment frameworks. The list is organized by how a practitioner usually works: choose a modality, pick datasets, select guard baselines, test against attacks, and wire the guard into a runtime system.

Contributions are welcome. If you find missing papers, datasets, models, or tools, please open an issue or submit a pull request.

Contents

• Benchmarks & Datasets
  • Text Safety
  • Multimodal Safety
  • Dialogue Safety
• Guard Models and Papers
  • LLM Guards
  • VLLM/MLLM Guards
• Attacks
  • White-box Attacks
  • Black-box Attacks
  • Red Teaming Tools
• Tools
  • Guardrail Frameworks
  • Evaluation Platforms
• Open Source Projects

Sorting: Tables with Venue & Year are sorted by year in descending order within each category.

---

📊 Benchmarks & Datasets

Modality Legend: [T] Text | [I] Image | [V] Video | [M] Multi-modal | [D] Dialogue

Text Safety

Benchmark	Paper	Venue & Year	Modality	Description / Highlights	Links
SafePyramid	SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing	arXiv 2026	[T],[D]	Hierarchical in-context policy guardrailing benchmark with 1K multi-turn conversations, 3K application-specific policies, and 61K+ natural-language rules across three difficulty levels	-
Gate AI Eval Harness	Gate AI: LLM Security Benchmark Evaluation Methodology and Results	arXiv 2026	[T]	Evaluation harness for prompt-injection and jailbreak detectors across 16 public benchmarks with global operating-point selection	-
StreamSafe	SentGuard: Sentence-Level Streaming Guardrails for Large Language Models	arXiv 2026	[T]	Sentence-level streaming safety benchmark with structured annotations across 8 harm categories and risk evolution over reasoning/response segments	-
GuardZoo	Triaging Threats to Specialized Guardrails	arXiv 2026	[T]	Human-annotated guardrail benchmark with 32K+ samples across 15 unsafe categories	-
PII-Bench	GLiNER Guard: Unified Encoder Family for Production LLM Safety and Privacy	arXiv 2026	[T]	Span-level PII benchmark for end-to-end privacy detection in guardrail pipelines	Dataset
ATBench	AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security	arXiv 2026	[T]	Fine-grained agentic safety benchmark with risk source, failure mode, and consequence taxonomy	Project
ExpGuardMix	ExpGuard: LLM Content Moderation in Specialized Domains	ICLR 2026	[T]	Domain-specific moderation data for finance, medical, and legal safety	Dataset
Aegis2	AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails	NAACL 2025	[T]	Diverse safety data	Dataset
Nemotron Safety Guard Dataset v3	Llama Nemotron Safety Guard: A Multilingual Input-Output Safety Model and Reasoning Dataset	arXiv 2025	[T]	Multilingual guard training data, 500K+ samples across 12 languages	Dataset
PolyGuard	PolyGuard: Towards Detecting Unsafe Multilingual LLM Prompts	COLM 2025	[T]	Multilingual safety	Dataset
SocialHarmBench	SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests	arXiv 2025	[T]	Sociopolitical harms (585 prompts, 34 countries)	Dataset
AgentHarm	AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents	ICLR 2025	[T]	LLM agent harmfulness (260 behaviors)	Dataset
Pre-Exec Bench	Building a Foundational Guardrail for General Agentic Systems via Synthetic Data	arXiv 2025	[T]	Pre-execution agent safety benchmark for detection, categorization, explanation, and cross-planner generalization	-
HarmBench	HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal	ICML 2024	[T]	Automated red teaming benchmark	Dataset
JailbreakBench	JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models	NeurIPS 2024	[T]	Jailbreak robustness evaluation	Dataset
WildChat	WildChat: 1M ChatGPT Interaction Logs in the Wild	ICLR 2024	[T]	Real-world jailbreak dataset (1M conversations)	Dataset
LMSYS-Chat-1M	LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset	ICLR 2024	[T]	Chatbot arena dataset	Dataset
SafetyBench	SafetyBench: Evaluating the Safety of Large Language Models	ACL 2024	[T]	Comprehensive evaluation (11K prompts)	Dataset
XSTest	XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models	NAACL 2024	[T]	Refusal on safe prompts (200+)	Dataset
SafeRLHF	Safe RLHF: Safe Reinforcement Learning from Human Feedback	ICLR 2024	[T]	Multi-level alignment	Dataset
WildGuardMix	WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs	NeurIPS 2024	[T]	Mixture of safety data	Dataset
StrongREJECT	StrongREJECT: A Strong Reject for Empty Jailbreaks	NeurIPS 2024	[T]	Jailbreak detection (313 prompts)	Dataset
Do-Not-Answer	Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs	ACL 2024	[T]	Evaluation of refusals	Dataset
Jailjudge	JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework	arXiv 2024	[T]	Comprehensive jailbreak judge	Dataset
ToxicChat	ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation	EMNLP 2023	[T]	Real-world toxicity detection	Dataset
BeaverTails	BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset	NeurIPS 2023	[T]	Safety pairs (360K pairs)	Dataset

Multimodal Safety

Benchmark	Paper	Venue & Year	Modality	Description / Highlights	Links
EgoSafetyBench	EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards	arXiv 2026	[V],[M]	Egocentric video benchmark with 1,200 robot-view scenarios and half-second annotations for streaming embodied VLM safety guards	-
SingGuard-Bench	SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning	arXiv 2026	[M],[D]	Multimodal guardrail benchmark with 56K+ examples, 80+ fine-grained risk types, dynamic-rule evaluation, and cross-modal joint-risk cases	Code
MTMCS-Bench	MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues	arXiv 2026	[M],[D]	Multi-turn multimodal contextual safety benchmark with escalation and context-switch risk settings	Dataset Code
VLSU	VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety	ICLR 2026	[M]	Vision Language Safety Understanding (8K+ pairs)	-
SafeEditBench	Towards Policy-Adaptive Image Guardrail: Benchmark and Method	arXiv 2026	[I],[M]	Cross-policy image guardrail benchmark with policy-aligned safe/unsafe image pairs	Dataset
SafeVision	SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability	arXiv 2025	[M]	Multimodal safety evaluation	-
SafeWatch	SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations	ICLR 2025	[V]	Video guardrail (2M+ videos)	-
VisionHarm	SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability	arXiv 2025	[I]	Image safety dataset used by SafeVision	-
UnsafeBench	UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images	CCS 2025	[I]	Image classifier safety	-
MSTS	MSTS: A Multimodal Safety Test Suite for Vision-Language Models	arXiv 2025	[M]	Multimodal Safety Test Suite	-
MSSBench	Multimodal Situational Safety	ICLR 2025	[M]	Multimodal situational safety	-
Video-SafetyBench	Video-SafetyBench: A Benchmark for Safety Evaluation of Video Large Language Models	NeurIPS 2025	[V]	Video safety benchmark	-
SafeMT	SafeMT: Multi-turn Safety for Multimodal Language Models	arXiv 2025	[M],[D]	Multi-turn multimodal safety	-
BeaverTails-V	Safe RLHF-V	arXiv 2025	[I],[T]	Multimodal safety data across visual/text harm categories	Dataset
MM-SafetyBench	MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models	ECCV 2024	[M]	Multimodal safety benchmark	-
JailbreakV-28K	JailbreakV-28K: A Benchmark for Assessing the Robustness of Multimodal Large Language Models against Jailbreak Attacks	arXiv 2024	[M]	28K jailbreak pairs	-

Dialogue Safety

Benchmark	Paper	Venue & Year	Modality	Description / Highlights	Links
SafeDialBench	SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks	arXiv 2025	[D]	Multi-turn jailbreak dialogues	-
CoSafe	CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference	EMNLP 2024	[D]	Multi-turn dialogue safety	-

---

🛡️ Guard Models and Papers

LLM Guards

Model	Paper	Venue & Year	Modality	Description / Highlights	Links
DT-Guard	DT-Guard: Intent-Driven Reasoning-Active Training for Reasoning-Free LLM Safety Guardrail	arXiv 2026	[T]	Reasoning-active training and reasoning-free inference guardrail that internalizes intent, category, and safety labels for low-latency moderation	-
HaloGuard 1.0	HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety	arXiv 2026	[T]	Open-weights constitutional classifier for multilingual prompt safety across 46 policies and 46 languages	HF
kNNGuard	kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail	arXiv 2026	[T]	Training-free configurable guardrail using hidden activations and small safe/unsafe prompt banks for fast domain adaptation	-
LeanGuard	Do Safety Guardrails Need to Reason? LeanGuard: A Fast and Light Approach for Robust Moderation	arXiv 2026	[T]	Lightweight label-only encoder guardrail that questions CoT necessity and reports ~100x lower inference compute than reasoning guards	Code
TRIAD	From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents	arXiv 2026	[T]	Guardrail-integrated agent framework that turns safety feedback into plan remediation instead of only allow/block decisions	Code
IndicGuard	IndicGuard: A Multilingual Safety Guard Model and Dataset for Indic Languages	arXiv 2026	[T]	Multilingual safety guard model and culturally nuanced dataset for ten major Indic languages, localized harm categories, and adversarial jailbreaks	-
Membrane	Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defense	arXiv 2026	[T]	Self-evolving guardrail using Contrastive Safety Memory for adaptive jailbreak and agent-safety defense without retraining	-
GuardNet	GuardNet: Ensemble Strategies of Shallow Neural Networks for Robust Prompt Injection and Jailbreak Detection	arXiv 2026	[T]	Lightweight BiLSTM ensemble guardrail for low-latency prompt-injection and jailbreak detection	-
SentGuard	SentGuard: Sentence-Level Streaming Guardrails for Large Language Models	arXiv 2026	[T]	Parallel sentence-level streaming guardrail that buffers generated text and moderates complete sentence chunks before release	-
BraveGuard	BraveGuard: From Open-World Threats to Safer Computer-Use Agents	arXiv 2026	[T]	Self-evolving trajectory-level guard training from open-world threat signals for computer-use agents	HF
ConsisGuard	ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails	arXiv 2026	[T]	Consistency-aware reasoning guardrail that aligns policy-grounded deliberation with final safety decisions	-
RouteGuard	Triaging Threats to Specialized Guardrails	arXiv 2026	[T]	Router-expert guardrail framework that dispatches conversations to specialized threat-domain experts	-
CoLaGuard	Robust and Efficient Guardrails with Latent Reasoning	arXiv 2026	[T]	Latent-reasoning guardrail that internalizes safety rationales for lower-latency prompt and response moderation	-
GLiNER Guard	GLiNER Guard: Unified Encoder Family for Production LLM Safety and Privacy	arXiv 2026	[T]	Unified encoder family for safety moderation, PII detection, and prompt attack detection in one forward pass	HF Dataset
GLiGuard	GLiGuard: Schema-Conditioned Classification for LLM Safeguard	arXiv 2026	[T]	Compact schema-conditioned bidirectional encoder for prompt safety, response safety, refusals, harm categories, and jailbreak strategies	HF Code
FlexGuard	FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation	ACL 2026	[T]	Continuous risk scoring	HF Code
SafeDream	SafeDream: Safety World Model for Proactive Early Jailbreak Detection	arXiv 2026	[T]	Safety alignment via dream methodology	-
MOSAIC	MOSAIC: Composable Safety Alignment with Modular Control Tokens	arXiv 2026	[T]	Multi-dimensional safety analysis	-
YuFeng-XGuard	YuFeng-XGuard: A Reasoning-Centric, Interpretable, and Flexible Guardrail Model for Large Language Models	arXiv 2026	[T]	Tiered inference strategy with dynamic policy adjustments	HF
GaaA	Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs	arXiv 2026	[T]	Soft-gating pipeline where the guardrail acts as an advisor	-
LEG	A Lightweight Explainable Guardrail for Prompt Safety	ACL 2026	[T]	Modular external guardrail providing interpretable explanations	-
BARRED	BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debate	arXiv 2026	[T]	High-fidelity synthetic training data via multi-agent debate	-
SafeHarbor	SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety	ICML 2026	[T]	Training-free hierarchical memory-augmented guardrail for LLM agents with dynamic rule injection and self-evolving memory structure	Code
ToolSafe / TS-Guard	ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback	arXiv 2026	[T]	Step-level proactive guardrail for unsafe tool invocation in LLM agents	Code
SafePred	SafePred: A Predictive Guardrail for Computer-Using Agents via World Models	arXiv 2026	[T]	Predictive guardrail for computer-using agents with short- and long-term risk prediction	Code
SIREN	LLM Safety From Within: Detecting Harmful Content with Internal Representations	arXiv 2026	[T]	Lightweight harmfulness detector using safety neurons and adaptive layer-weighted internal representations	HF Code
AgentDoG	AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security	arXiv 2026	[T]	Diagnostic guardrail for agent trajectories with fine-grained risk taxonomy and root-cause explanations	Project
ExpGuard	ExpGuard: LLM Content Moderation in Specialized Domains	ICLR 2026	[T]	Specialized guardrail for harmful prompts and responses in finance, medical, and legal domains	Code Dataset Models
Llama Guard 4	-	Meta 2025	[M]	Multimodal safety model for image and text moderation	HF
Prompt Guard 2	-	Meta 2025	[T]	Malicious prompt injection detection	HF
Nemotron Safety Guard	-	NVIDIA 2025	[T]	Multilingual content safety guard trained with Nemotron/Aegis data	HF
Qwen3Guard-Gen	Qwen3Guard Technical Report	arXiv 2025	[T]	Generative guard with safety levels, categories, multilingual support	HF 0.6B HF 4B HF 8B Code
Qwen3Guard-Stream	Qwen3Guard Technical Report	arXiv 2025	[T]	Token-level streaming safety monitoring for incremental generation	HF 0.6B HF 4B HF 8B Code
Stable Guard	-	Stability AI 2025	[T]	Stability AI's safety model	HF
GuardReasoner	GuardReasoner: Towards Reasoning-based LLM Safeguards	arXiv 2025	[T]	Reasoning-based safeguard framework	Code
C-SafeGen	C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails	NeurIPS 2025	[T]	Certified safe generation via claim-based streaming	-
R2-Guard	R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning	ICLR 2025	[T]	Robust reasoning guardrail using prob-graphical models	-
Safiron	Building a Foundational Guardrail for General Agentic Systems via Synthetic Data	arXiv 2025	[T]	Foundational pre-execution guardrail for general agentic systems trained with synthetic trajectories	-
AGrail	AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection	ACL 2025	[T]	Lifelong agent guardrail with adaptive safety check generation and test-time refinement	Code
RoboGuard	Safety Guardrails for LLM-Enabled Robots	arXiv 2025	[T]	Two-stage guardrail for robot plans using grounded safety rules and temporal logic synthesis	Project
ShieldAgent	ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning	arXiv 2025	[T]	Guardrail agent using verifiable policy reasoning over agent action trajectories	Project
Llama Guard 3	-	Meta Connect 2024	[M]	Multimodal input/output guard	HF
Llama Guard 2	-	Meta 2024	[T]	Enhanced version	HF
ShieldGemma	ShieldGemma: Generative AI Content Moderation Based on Gemma	arXiv 2024	[T]	Google's safety filter	HF
WildGuard	WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs	NeurIPS 2024	[T]	One-stop moderation tool	HF Code
Aegis	AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts	arXiv 2024	[T]	Online adaptive content safety	HF
Shield	-	NVIDIA 2024	[T]	Safety description guided generation	HF Code
Granite Guardian	-	IBM 2024	[T]	Specialized open-source safety models	HF Code
Llama Guard	Llama Guard: Safeguarding Large Language Models	arXiv 2023	[T]	Pioneer guard model	HF Code
WildGuard Lite	-	-	[T]	Lightweight moderation model	HF
Beaver	-	PKU	[T]	Open source safety alignment	HF Code
GPT-4 Content Mod.	-	OpenAI	[T]	Built-in content moderation	API
Claude 3 Haiku	-	Anthropic	[T]	Constitutional AI based safety	API
Gemini Safety	-	Google	[T]	Built-in safety filters	API
Azure Content Safety	-	Microsoft	[T],[I]	Enterprise content moderation	API

VLLM/MLLM Guards

Model	Paper	Venue & Year	Modality	Description / Highlights	Links
SingGuard	SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoning	arXiv 2026	[M],[D]	Policy-adaptive multimodal guardrail family with runtime policy input, rule-level triggering, and fast/hybrid/slow reasoning regimes	Code
GuardReasoner-Omni	GuardReasoner-Omni: A Reasoning-based Multi-modal Guardrail for Text, Image, Video, and Audio	arXiv 2026	[M]	Omni-modal reasoning guardrail for text, image, video, and audio moderation with SFT and RL training	HF 3B HF 7B
SafeLens	SafeLens: Deliberate and Efficient Video Guardrails with Fast-and-Slow Screening	arXiv 2026	[V]	Video guardrail framework with fast-and-slow screening, influence-filtered SafeWatch data, and test-time reasoning traces	-
VLSU	VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety	ICLR 2026	[M]	Vision Language Safety Understanding (8K+ pairs)	Code
SafeGuard-VL	Towards Policy-Adaptive Image Guardrail: Benchmark and Method	arXiv 2026	[I],[M]	Policy-aware visual safety guardrail trained with RLVR/GRPO for dynamic safety policies	HF Dataset
HomeGuard	HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task	arXiv 2026	[M]	Embodied VLM safeguard for contextual household risk with grounded visual anchors	Code
Pragma-VL	Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs	ICLR 2026	[M]	End-to-end MLLM safety-helpfulness arbitration with risk-aware visual perception	Code Dataset
UniMod	From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderation	arXiv 2026	[M]	Multimodal moderation via dense reasoning trajectories, evidence grounding, risk mapping, and policy decisions	-
SafeVision	SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability	ICLR 2025	[M]	Multimodal safety evaluation	HF
SaFeR-VLM	SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models	arXiv 2025	[M]	Safety-aware multimodal reasoning framework using safety rollouts, structured rewards, and GRPO	Code
SafeWatch	SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations	ICLR 2025	[V]	Video guardrail model (2M+ videos)	HF Code
ShieldGemma 2	ShieldGemma 2: Robust and Tractable Image Content Moderation	arXiv 2025	[I],[M]	Open image moderation model for sexually explicit, violence/gore, and dangerous content	HF
Llama Guard 4	-	Meta 2025	[M]	Multimodal text/image guard for prompt and response moderation	HF
LlavaGuard	LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Models	ICML 2025	[I],[M]	VLM safeguard with safety rating, category, and rationale annotations	Code
GuardReasoner-VL	GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning	NeurIPS 2025	[M]	Reinforced reasoning-based guard model for VLM safety	Code
OMNIGUARD	OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Modalities	EMNLP 2025	[M]	Efficient safety moderation across languages and modalities using internal representations	Paper
VisionHarm	SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability	ICLR 2025	[I]	Image safety detection	-
LLaVAShield	LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models	arXiv 2025	[M],[D]	Safeguarding multimodal multi-turn dialogues	HF
UnsafeBench	UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images	CCS 2025	[I]	Image classifier safety evaluation	HF Code
MSTS	MSTS: A Multimodal Safety Test Suite for Vision-Language Models	arXiv 2025	[M]	Multimodal Safety Test Suite	HF Code
MSSBench	Multimodal Situational Safety	ICLR 2025	[M]	Multimodal situational safety	Code
Llama Guard 3 Vision	Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations	arXiv 2024	[M]	Multimodal input/output guard	HF Code
MM-SafetyBench	MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models	ECCV 2024	[M]	Multimodal safety benchmark	HF Code

---

⚔️ Attacks

White-box: Requires model weights/gradients | Black-box: API-only access

White-box Attacks

Attack	Paper	Venue & Year	Modality	Description / Highlights	Links
SABER	SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection	EMNLP 2025	[T]	Cross-layer residual injection bypasses safety mechanisms concentrated in middle-to-late layers	Code
GCG	Universal and Transferable Adversarial Attacks on Aligned Language Models	ICLR 2024	[T]	Gradient-based suffix optimization	Code
AutoDAN	AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models	ICLR 2024	[T]	Genetic algorithm for stealthy prompts	Code
COLD	COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability	ICML 2024	[T]	Controllable decoding attack	Code
AmpleGCG	AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs	COLM 2024	[T]	Universal transferable suffixes	Code
AdvPrompter	AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs	ICML 2024	[T]	Adversarial suffix generation	Code
MAC	Boosting Jailbreak Attack with Momentum	ICLR Workshop 2024	[T]	Momentum-based attack	Code
Harmful Fine-tuning	Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey	arXiv 2024	[T]	Harmful Fine-tuning Attacks and Defenses	-

Black-box Attacks

Attack	Paper	Venue & Year	Modality	Description / Highlights	Links
Guardrail Recon	Behind the Refusal: Determining Guardrail Activation via Behavioral Monitoring	arXiv 2026	[T]	Black-box reconnaissance method that distinguishes guardrail blocks from model refusals using HTTP, lexical, and timing signals	-
Guardrail DoS	From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrails	arXiv 2026	[T]	Availability attack that traps reasoning-based agent guardrails in extended reasoning loops, causing large token and latency amplification	-
Posterior Attack	Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack	arXiv 2026	[T]	Single-query jailbreak that exploits models' internal safety awareness to bypass guardrails	-
Prompt Overflow	Prompt Overflow: What the Guardrail Inspects Is Not What the Model Infers	arXiv 2026	[T]	Long-context attack that exploits mismatch between guardrail inspection windows and downstream LLM inference windows	-
JRS-Rem	Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift	arXiv 2026	[M]	Explores representation shift in VLM jailbreaks & defense	-
ALERT	ALERT: Zero-shot LLM Jailbreak Detection via Internal Discrepancy Amplification	arXiv 2026	[T]	Zero-shot detection via internal feature discrepancies	-
Echo Chamber	The Echo Chamber Multi-Turn LLM Jailbreak	arXiv 2026	[T],[D]	Multi-turn gradual escalation attack	-
AMIS	Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges	ICLR 2026	[T]	Meta-optimization evolving prompts and scoring	-
Odysseus	Odysseus: Jailbreaking Commercial Multimodal LLM-integrated Systems via Dual Steganography	NDSS 2026	[M]	Hides malicious intent via dual steganography in images/audio	-
PAIR	Jailbreaking Black Box Large Language Models in Twenty Queries	SaTML 2025	[T]	Iterative LLM-based refinement	Code
FigStep	FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts	AAAI 2025	[M]	VLM multimodal jailbreak	-
Crescendo	Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack	USENIX 2025	[T],[D]	Multi-turn escalating attack	-
X-Teaming	X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents	COLM 2025	[T],[D]	Multi-agent coordination	-
FlipAttack	FlipAttack: Jailbreak LLMs via Flipping	ICML 2025	[T]	Left-side noise attack	Code
E2AT / ProEAT	E$^2$AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models	arXiv 2025	[M]	Adversarial training paradigm tailored for MLLMs	-
LogiBreak	Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression	arXiv 2025	[T]	Converts prompts to formal logical expressions	-
DeepInception	DeepInception: Hypnotize Large Language Model to Be Jailbreaker	NeurIPS 2024	[T]	Multi-layer role-play	Code
TAP	Tree of Attacks: Jailbreaking Black-Box LLMs Automatically	NeurIPS 2024	[T]	Tree-of-thought attack	Code
ArtPrompt	ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs	ACL 2024	[T]	ASCII art obfuscation	Code
SelfCipher	GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher	ICLR 2024	[T]	Cipher encoding	Code
ReNeLLM	A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily	ACL 2024	[T]	Disguise & reconstruction	Code
Many-shot	Many-shot Jailbreaking	Anthropic 2024	[T]	Long-context few-to-many-shot jailbreaking	-
FuzzLLM	FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models	ICASSP 2024	[T]	Fuzzing-based discovery	Code
MasterKey	MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots	NDSS 2024	[T]	Chatbot auto-jailbreak	-
CodeChameleon	CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models	ACL 2024	[T]	Personalized encryption	Code
SOP	SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters	NeurIPS W 2024	[T]	Social facilitation attack	Code

Red Teaming Tools

Tool	Paper	Links	Description
Garak	-	Code	LLM vulnerability scanner
JailbreakBench	-	Code	Robustness benchmark
MMRT	-	-	Multimodal multi-turn red teaming
Inspectus	-	-	Security evaluation
Aporia	-	-	LLM monitoring and safety

---

🧰 Tools

Guardrail Frameworks

Tool	Paper	Links	Stars	Description
Armorer Guard	-	Code	⭐40+	Local Rust scanner and MCP proxy for AI-agent prompt injection, credential leaks, exfiltration, and risky tool calls
NeMo Guardrails	-	Code	⭐6K+	NVIDIA's programmable guardrails
LLM-Guard	-	Code	⭐2.8K+	Content filtering library
Guardrails	-	Code	⭐6.7K+	Policy enforcement
DarkMoon Privacy Gateway	-	Code	-	Self-hosted proxy and MCP host that tokenizes sensitive values into deterministic reversible placeholders before prompts reach the LLM, with an anti-exfiltration command gateway
Promptfoo	-	Code	⭐20K+	LLM testing and safety evaluation
gate.cat	-	Code	⭐0	Deterministic, fail-closed action veto that blocks irreversible shell commands (rm -rf, DROP TABLE, terraform destroy) from AI coding agents before they run, with no LLM in the veto path; runs as a Claude Code PreToolUse hook, gated shell, or local OpenAI-API proxy
verbatim-citation-gate	-	Code	⭐3	Deterministic, fail-closed citation check that verifies a quoted span literally appears in the source it was attributed to, returning found/misattributed/not_found with no LLM in the verification path; Unicode-aware normalization, usable as a pre-filter ahead of a judge model

Evaluation Platforms

Tool	Paper	Description
LM Eval Harness	-	Standardized evaluation
Confident AI	-	LLM evaluation platform
Giskard	-	Testing for ML models
Helicone	-	LLM observability
Guardrails Hub	-	Pre-built guardrail policies

---

🚀 Open Source Projects

Project	Paper	Description	Links	Stars
ArmorerLabs/Armorer-Guard	-	Local Rust scanner and MCP proxy for AI-agent security	GitHub	⭐40+
enguard-ai/awesome-ai-guardrails	-	Curated AI guardrails resources	GitHub	⭐50+
beyefendi/awesome-llm-security	-	LLM security resources	GitHub	⭐10+
yueliu1999/Awesome-Jailbreak-on-LLMs	-	Jailbreak resources	GitHub	⭐1.3K+
CryptoAILab/Awesome-LM-SSP	-	Safety security privacy	GitHub	⭐1.9K+
NVIDIA/NeMo-Guardrails	-	Programmable guardrails	GitHub	⭐6K+
laiyer-ai/LLM-Guard	-	Content filtering	GitHub	⭐2.8K+
guardrails-ai/guardrails	-	Policy enforcement	GitHub	⭐6.7K+
promptfoo/promptfoo	-	LLM testing	GitHub	⭐20K+
NVIDIA/garak	-	LLM vulnerability scanner	GitHub	⭐7.6K+
allenai/wildguard	-	Safety moderation	GitHub	⭐100+
PKU-Alignment/safe-rlhf	-	Safety alignment	GitHub	⭐1.6K+

---

---

✉️ Contact Us

For questions or collaborations, please contact:

• Zongyi Li: lizongyi.lzy@antgroup.com
• Yichen Bai: baiyichen.byc@antgroup.com
• Liangbo He: liangbo.hlb@antgroup.com
• Jun Lan: yelan.lj@antgroup.com