A curated list of open-source AI-assisted penetration testing tools, frameworks, CTF agents, and benchmarks.
Compiled from a systematic review of the field, anchored by the analysis in "What Makes a Good LLM Agent for Real-world Penetration Testing?" (Deng et al., 2025 — arXiv:2602.17622), which surveyed 28 LLM-based pentesting systems and identified two failure categories: Type A (capability gaps) and Type B (complexity barriers).
- Pentesting Agents & Frameworks
- CTF-Focused Agents
- Benchmarks & Evaluation
- Related Resources
- Key Concepts from the Paper
- Contributing
These systems were directly analysed or benchmarked in Deng et al. (2025). The paper evaluated five of them (marked with results) across XBOW, PentestGPT-Benchmark, and GOAD.
| Tool | Architecture | Year | Paper Results (GPT-5) | Links |
|---|---|---|---|---|
| PentestGPT | Human-in-the-loop copilot, task tree planning | 2024 | XBOW 42%, 7/13 machines, 1/5 hosts | GitHub · Paper |
| AutoPT | Single-agent, pentesting state machine | 2024 | XBOW 40%, 7/13 machines, 1/5 hosts | Paper |
| PentestAgent | Multi-agent with RAG, phase-based planning | 2024 | XBOW 49%, 7/13 machines, 1/5 hosts | Paper |
| VulnBot | Multi-agent, tri-phase planning | 2025 | XBOW 45%, 8/13 machines, 1/5 hosts | Paper |
| Cochise | Multi-agent, AD-specialised, hierarchical | 2025 | XBOW 43%, 4/13 machines, 2/5 hosts | Paper |
| PENTESTGPT V2 | Tool Layer + TDA-EGATS + Memory | 2025 | XBOW 85%, 12/13 machines, 4/5 hosts | Paper · Code |
| RapidPen | Single-agent, ReAct with RAG | 2025 | Surveyed (not benchmarked) | Paper |
| xOffense | Multi-agent, fine-tuned knowledge | 2024 | Surveyed (not benchmarked) | Paper |
| TermiAgent | Multi-agent, memory tree | 2024 | Surveyed (not benchmarked) | Paper |
| HackSynth | Single-agent, evaluation framework | 2024 | Surveyed (not benchmarked) | Paper |
| MAPTA | Multi-agent, web pentesting | 2025 | Surveyed (not benchmarked) | Paper |
Open-source tools and frameworks discovered beyond the paper's survey.
| Tool | Description | Links |
|---|---|---|
| CAI | Cybersecurity AI framework by Alias Robotics. REPL-based pentesting with Phoenix/OTel tracing, JSONL recording, and CAIBench evaluation. | GitHub |
| MANTIS | Multi-agent pentesting framework on smolagents. Structured state store, HITL operator shell, typed security tool wrappers, OpenTelemetry tracing. | GitHub |
| Shannon | Autonomous AI hacker achieving 96.15% on XBOW (hint-free, source-aware). Hunts for attack vectors in code and uses a built-in browser to execute real exploits. | GitHub |
| PentAGI | Fully autonomous AI agent system with 20+ built-in security tools. Supports OpenAI, Anthropic, Ollama, AWS Bedrock, Google AI, DeepSeek. | GitHub |
| hackingBuddyGPT | Framework for LLM-driven security testing. Includes "Wintermute" for Linux privilege escalation via SSH. GitHub Accelerator 2024, presented at FSE'23. | GitHub |
| Strix | Graph-based parallel agent workflows for pentesting. Assigns specialised agents to different targets, includes HTTP proxy, browser automation, and terminal environments. | GitHub |
| NeuroSploit | AI pentesting framework (v3) with 100 vulnerability types across 10 categories. 3-stream parallel pentesting, isolated Kali Docker containers, anti-hallucination pipeline. | GitHub |
| Redamon | AI-powered red team framework automating recon to post-exploitation. Neo4j graph database as single source of truth, LangGraph-based autonomous agent, MCP tool selection. | GitHub |
| HexStrike AI | MCP server enabling AI agents (Claude, GPT, etc.) to run 150+ cybersecurity tools autonomously. Bridges LLMs with real-world offensive security capabilities. | GitHub |
| Pentest Copilot | Browser-based AI hacking assistant with integrated Kali Linux container, VPN integration, chain-of-thought reasoning, and RAG. | GitHub |
| Reaper | Go-based application security testing framework by Ghost Security. MITM HTTPS proxy for AI agents with autonomous attack type selection. Apache-2.0. | GitHub |
| AutoPentester | LLM agent framework with RAG, repetition identifier (90.5% loop reduction), and results verifier. 27% better subtask completion than PentestGPT. | GitHub · Paper |
| AutoPentest | CLI tool for autonomous black-box penetration tests using GPT-4o and LangChain. Automated feedback loop with optional human review. | GitHub · Paper |
| Nebula | AI-powered pentesting assistant for recon, note-taking, and vulnerability analysis. Integrates Llama-3.1, Mistral-7B, DeepSeek-R1, and OpenAI. BSD-2-Clause. | GitHub |
| AI-OPS | Pentesting AI assistant based on open-source LLMs. Focuses on exploit development, vulnerability research, and code analysis. | GitHub |
| Floki | Agentic workflow framework for cybersecurity built on Dapr. Service-to-service invocation, pub/sub messaging, workflow API, and state management. | GitHub |
| TARS | AI-powered pentesting tool that builds agents using existing cybersecurity tools for vulnerability scanning and real-time countermeasure generation. | GitHub |
| Decepticon | Autonomous multi-agent red team testing service built on LangChain/LangGraph. | GitHub |
| Tool | Description | Links |
|---|---|---|
| D-CIPHER | Multi-agent CTF framework (Planner + Executors + Auto-prompter). SOTA: 22.0% NYU CTF Bench, 22.5% Cybench, 44.0% HackTheBox. | GitHub · Paper |
| CTFAgent | Plan-and-execute agent with stateful task tree, two-stage RAG, and environmental augmentation. Surpasses ~94% of CTF teams in HITL mode. Top 23.6% in picoCTF2024. | Paper (ScienceDirect) |
| Cyber-Zero | Amazon Science. Trains cybersecurity agents from CTF writeups without runtime environments. Cyber-Zero-32B matches DeepSeek-V3 and Claude 3.5 Sonnet. | GitHub · Paper |
| InterCode-CTF Agent | Demonstrates 95% on InterCode-CTF using ReAct&Plan prompting with expanded toolsets. Saturated this benchmark, pushing the field to harder evaluations. | GitHub · Paper |
| llm-ctf-agent | Boilerplate for developing and evaluating LLM-based CTF agents. Framework for running agents against challenges, managing LLM interactions, and logging. | GitHub |
| Benchmark | Description | Links |
|---|---|---|
| XBOW | 104 web security CTF tasks (SQLi, XSS, auth bypass). Used to evaluate Type A failures. Commercial platform. | Website |
| PentestGPT Benchmark | 13 HTB/VulnHub machines (Easy to Hard), 9-22 subtasks each. End-to-end pentesting evaluation. | Paper |
| GOAD | 5-host multi-domain Active Directory lab. Requires credential chaining, Kerberoasting, lateral movement. Tests Type B failures. | GitHub |
| AutoPenBench | Docker-based containerised vulnerable environments with milestone-based metrics. Synthetic + real-world CTF-style tasks. | GitHub · Paper |
| Cybench | 40 professional-level CTF tasks from 4 competitions across 6 domains. Used by US AISI and UK AISI for pre-deployment model testing. | GitHub · Paper |
| NYU CTF Bench | 200 dockerised CSAW CTF challenges across 6 categories (web, pwn, forensics, rev, crypto, misc). | GitHub · Paper |
| CAIBench | Meta-benchmark integrating 10,000+ instances across 5 evaluation categories including Jeopardy CTFs, Attack/Defence CTFs, and Cyber Range exercises. | Paper |
| 3CB | 15 original CTF challenges covering all MITRE ATT&CK categories. Novel challenges absent from model training data. Used by UK AISI. | Website · Paper |
| CTFusion | Streaming evaluation framework built on live CTFs to prevent data contamination. Implemented as MCP server on CTFd platform. | Paper |
| HackTheBox AI Range | Commercial platform for continuous real-world AI agent evaluation against OWASP Top 10 (2025). | Website |
| AI-Pentest-Benchmark | Community benchmark for evaluating pentesting AI. | GitHub |
| Resource | Description | Links |
|---|---|---|
| awesome-cybersecurity-agentic-ai | Curated list of agentic AI frameworks and cybersecurity tools. | GitHub |
| Awesome-LLM4Cybersecurity | Overview of LLMs for cybersecurity covering offensive and defensive applications. | GitHub |
| AI-agents-for-cybersecurity | Course materials for AI Agents and RAG for Cybersecurity Operations by Omar Santos. | GitHub |
The paper provides a framework for understanding why AI pentesting agents succeed or fail, which is useful context for evaluating any tool on this list.
- Type A failures (42%) — Capability gaps: missing tools, incorrect syntax, knowledge gaps. Addressed by better tool interfaces, RAG, and engineering.
- Type B failures (58%) — Complexity barriers: context forgetting (18%), premature commitment (16%), exploration-exploitation imbalance (12%), multi-step chain failures (12%). Require architectural solutions.
- Tool & Skill Layer — 38 typed tool interfaces with input schemas, structured output parsing, pre/postconditions. Skill composition for multi-tool attack patterns. +14% on XBOW in ablation.
- Task Difficulty Assessment (TDA) — Four dimensions: horizon estimation, evidence confidence, context load, historical success rate. Drives exploration vs exploitation decisions.
- Evidence-Guided Attack Tree Search (EGATS) — UCB node selection with difficulty penalties, TDI-guided mode switching, evidence-based pruning.
- Memory Subsystem — Structured state store (hosts, services, credentials, sessions, vulnerabilities), selective context injection, branch summaries.
- Creativity barrier — Novel exploitation requiring out-of-distribution reasoning
- Adversarial environments — Honeypots, canary tokens, deceptive services
- Temporal scale — Cross-session continuity for multi-week engagements
Contributions welcome. Please open a PR or issue to add tools, correct information, or suggest improvements. When adding a tool, include:
- Name and brief description
- GitHub link (if open-source)
- Paper link (if published)
- Whether it was evaluated on any standard benchmark
This list is released under CC0 1.0. The tools listed have their own licenses.