Repository navigation
Status
This document is honest about what is already production-ready and what is still at the research stage.
| Component | Status | Confidence |
|---|---|---|
| REST API (48 endpoints) | ✅ Production-ready | High |
| Multi-agent orchestration | ✅ Production-ready | Medium |
| PTES report format | ✅ Production-ready | High |
| ISO 27001 risk register | ✅ Production-ready | High |
| Plugin marketplace | ✅ Running | Medium |
| Session management | ✅ Production-ready | High |
| Scope enforcement + expansion workflow | ✅ Production-ready | High |
| Audit logging (tamper-evident) | ✅ Production-ready | High |
| Engagement brief + guardrails | ✅ Production-ready | High |
| Evidence vault (attach/verify/export) | ✅ Production-ready | High |
| Pending items tracker | ✅ Production-ready | High |
| HTML + Word (.docx) report | ✅ Production-ready | High |
| OWASP/CWE/CVSS4.0/EPSS tagging | ✅ Production-ready | High |
| Skill mode (junior/mid/advanced) | ✅ Production-ready | High |
| Tool parsers (Nmap/Gobuster/Hydra) | ✅ Running + verified | Medium |
| Exploit validation gate (validated vs candidate) | ✅ Implemented + unit-tested | High |
| Browser automation (Playwright) | ✅ Implemented · live smoke · shipped in image | Medium |
| SARIF 2.1.0 + PDF export | ✅ Implemented + unit-tested | Medium |
| CI gate (validated findings only) | ✅ Implemented | Medium |
| Run budget (time/tokens) | ✅ Implemented + unit-tested | High |
| MCP client (stdio JSON-RPC) | ✅ Implemented + unit-tested | Medium |
| Multi-LLM routing (15 providers) | ✅ Anthropic + Ollama live-verified · 11 cloud keys empty; 9router/custom not configured | Medium |
| ReAct agent loop + state machine | ⚠ Phase A: 4/20 · Clean 2026-10-24: 8/20 (40%), 0 harness errors · Strix 5-target subset: 3/5 vs native 3/5, union 4/5 | Medium |
| Kill chain orchestration | ⚠ Not live tested | Low |
| Business logic testing | ⚠ Not live tested | Low |
| LearningEngine | ❌ Disabled — see note | — |
2073 tests collected. Full suite on 2026-10-24: 2053 passed, 20 skipped in 2:14 (pytest -n 4). Skips are Docker-only host binaries and opt-in lab/HTB/Docker tests. See MILESTONES.md for M1–M3. Every component below has assertions that verify real behavior:
- REST API — 48 endpoints tested with real HTTP requests, RBAC, SSE streaming, session diff
- Multi-agent orchestrator — ReconSubAgent + WebSubAgent + InfraSubAgent + ReportSubAgent, parallel execution, phase gates per skill mode
- PTES report — JSON + Markdown, phase mapping, business impact, overall risk rating
- ISO 27001 risk register — Annex A control mapping, risk scoring, residual risk
- Plugin marketplace — search/install/verify checksum, GitHub registry
- Session management — save/load roundtrip, all fields including Finding, ServiceInfo, credentials
- Scope enforcement — CIDR, domain, wildcard, kill switch, expansion request/approve/deny
- Engagement brief — validate_action() for no_brute_force, production_systems, notify_before_exploit
- Evidence vault — attach/hash/verify/export ZIP bundle, tamper detection
- Pending tracker — detection of suspicious services (Samba 3.0.20, vsftpd 2.3.4, old SSH), untested endpoints, missing PoC
- Skill modes — junior/mid/advanced configs, RAG per mode, phase gates per mode
- Scope expansion — request → approve → audit log → added to scope
- Audit logging — hash chain, tamper detection
- Word report (.docx) — generate_docx.js available, Node.js wrapper verified
- Parsers — Nmap XML/text, Gobuster, Hydra with real output samples
Full suite after triage: 2053 passed, 20 skipped (pytest -n 4,
2073 collected). Two failures from the previous interrupted run were traced to
one root cause and fixed:
| Test | Cause | Fix |
|---|---|---|
test_coverage_boost.py::TestReActExtra::test_run_done_action_stops |
The flag-evidence stop guard ran for every objective, so a non-flag done action looped until the iteration cap |
_requires_flag_evidence(): the guard now applies only when the objective mentions a flag, or PENTESTAI_REQUIRE_FLAG=1
|
test_infra_mock.py::TestReActLoop::test_react_loop_stops_on_stop_signal |
Same root cause | Same fix, with a regression test in tests/test_m3_policy.py
|
Flag-capture benchmark objectives still reject done until flag{...} is in
the evidence. Regression coverage was also added for the XBEN-018 malformed
metadata guard in tests/test_react_discovery.py.
The SSE endpoint tests now clear provider env vars and local-service probes in their fixture. A running Ollama instance previously turned those tests into live calls and produced 60 s timeouts under xdist.
PTES reports now derive CVSS/OWASP/CWE from tag_finding() when a finding has
no stored compliance fields. Previously every PTES row rendered CVSS 0.0
(the viding.org report showed 0.0 for High findings); the new regression test is
in tests/test_fase6.py.
- Git repository initialized (
d2fcaf4, tagbaseline-2026-10-24);.env, sessions, logs, reports, evidence, and artifacts remain ignored. -
pentestai:latestrebuilt as a code layer (626e4cb8…) on the 2026-10-22 base, because a full Kali/MSF rebuild needs more free disk than this host has. It contains the stop-guard and PTES fixes. - XBEN-021 smoke on the new image: solved, 118.9 s; challenge image/volume cleaned afterwards. Disk free after cleanup: 11 GB.
- XSS verifier upgraded to browser-execution proof (DOM marker /
alert), with cookie support for authenticated pages. Live check on DVWA: confirmed. - Registration bootstrap: fills required
first_name/last_namefields and logs in with the new account after signup. Verified on XBEN-024 and DVWA. - Agent runtime layer:
agents/backend.pydefinesAgentBackend/NativeReActBackend;bench_agent --modelrecordsmodel_id,backend, andharness_version. Tolerant JSON parsing (core/json_repair.py) handles fences, trailing commas, single quotes, and truncated plans. -
Strix adapter (early M2):
agents/strix_backend.pyruns Strix OSS in headless mode, importsvulnerabilities.json/run.json, and adds findings as candidates. Live check: a Strix run on DVWA produced 1 XSS finding, imported with 1.46M tokens / $1.02 recorded. OSS Strix needs no account, only Docker +STRIX_LLM/LLM_API_KEY. -
Head-to-head (5 targets): native 3/5, Strix 3/5, union 4/5 in one
Strix-only run; XBEN-091's solve did not reproduce in three later runs.
Native won on GraphQL/speed/cost; Strix cost ~$4.70 and ~7.8M tokens for the
subset. Artifacts:
artifacts/bench/strix5/. -
Opt-in auto-SSRF (
PENTESTAI_AUTO_SSRF=1): after a URL/fetch form is discovered and an auth session exists, the loop runs the verified form-SSRF chain once using the last browser URL. Assisted 024 run: the chain triggered, discovered/reset_password, and created 1 validated SSRF finding, but did not capture the flag in that run (admin-login/reset state). The direct tool test on a fresh challenge does capture the flag. -
XBEN-024 SSRF verifier (2026-10-25): the
ssrftool now parses a profile/URL form, discovers documented localhost endpoints, sends a multipart POST with preserved form values, follows a documented/reset_passwordchain, logs in as admin, and captures the flag. Direct live test: solved with the flag in evidence. The autonomous benchmark run still does not pass auth cookies into the tool reliably; that planner/tool integration remains the gap, not the verifier. -
Native-first router:
agents/engine_router.pygives native 40% of the budget, then escalates to Strix only if the objective is not met. First five-target result: router 3/5, native 3/5, Strix 3/5; no solve-rate gain, ~$2.06 extra Strix cost. XBEN-091's earlier Strix solve did not reproduce in three runs, so the router is not default yet. Artifacts:artifacts/bench/router5/. - XBEN-024 SSRF chain verified with the real tools: register → auto-login →
profile_picture_url=http://127.0.0.1/reset_password?...→ admin password reset → login admin → flag captured. The autonomous re-run was cut short by LLM latency (60 s per reasoning call) under the 300 s budget; the tool chain itself works. - XBEN-018 is environment-blocked: its bundled checker reports
TypeError: Attempting to change the setter of an unconfigurable propertyfor benign input too (QtWebEngine), so it cannot reach its success condition. Excluding it, the clean run is 8/19 (42%); the headline stays 8/20.
The stack was rerun from scratch (Docker Desktop 4.96.0, Engine 29.9.1, Compose 5.6.0) after the host disk filled up and caused the daemon to hang (EOF).
| Item | Result |
|---|---|
pentestai-pentestai-1 |
Up, non-root user pentester
|
pentestai-neo4j-1 |
Up (healthy), Basic auth HTTP 200, bolt:7687 open |
pentestai-qdrant-1 |
Up, /collections HTTP 200 |
pentestai-msf-1 |
Up, MSGRPC ready on :55553 |
| Image |
pentestai:latest 6.27 GB, .env not baked in (7.83 GB after adding Chromium) |
| Injected env | 35 variables via env_file: .env
|
| Tools in image | nmap, sqlmap, nikto, gobuster, ffuf, hydra, whatweb, msfconsole, node, python3 |
scripts/env_test.py |
38/53 pass |
| LLM end-to-end |
build_router_from_env() → anthropic/claude-sonnet-4-6, response OK
|
The 15 FAILs in the smoke test come from empty .env parameters, not container defects:
OPENAI_API_KEY, XAI_API_KEY, MISTRAL_API_KEY, DEEPSEEK_API_KEY, GROQ_API_KEY, GOOGLE_API_KEY, PERPLEXITY_API_KEY, COHERE_API_KEY, TOGETHER_API_KEY, OPENROUTER_API_KEY, AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, TEST_TENANT_A, TEST_TENANT_B, TEST_OBJECT_ID.
Reproduction:
docker cp scripts/env_test.py pentestai-pentestai-1:/tmp/env_test.py
docker compose exec -T pentestai python3 /tmp/env_test.pyNot yet validated: live calls to the other cloud providers (their keys are empty), and the TEST_* login flow (credentials are intentionally not sent during the smoke test to avoid account lockout). Ollama was live-verified locally on 2026-10-24.
The gaps identified in the capability review were implemented. Code and unit tests are in place; the real-target run below is read-only, and no autonomous exploitation claim is made.
89 new unit tests were added and pass. The pre-existing full-suite run still
hits a timeout in this development environment; targeted suites
(test_api, test_shell, test_core, test_phase2, test_fase456,
test_gap_fixes, test_gaps) pass.
| Gap | Change | Verification |
|---|---|---|
| Findings could be speculative |
core/validation.py gate; validate command; Validation column in reports |
10 unit tests |
| No browser / JS support |
tools/browser.py (Playwright): open, click, fill, login, screenshot, eval |
8 unit tests with a mocked page |
| No machine-readable output |
reports/sarif.py (SARIF 2.1.0, validated-only by default) |
4 unit tests |
| No PDF |
reports/pdf.py (Chromium print-to-PDF) |
availability + error path |
| No CI gate |
scripts/ci_gate.py + .github/workflows/pentest-gate.yml
|
exit codes 0/2/3 |
| No benchmark harness |
scripts/bench_agent.py + BENCHMARK.md
|
XBOW pilot 0/5 · DVWA 33 findings |
| Runaway runs / cost |
PENTESTAI_MAX_SECONDS, PENTESTAI_MAX_TOKENS, ReActAgent budget |
5 unit tests |
| No MCP support |
tools/mcp.py + mcp list/tools/call/close commands |
9 unit tests + live secure-filesystem-server 2.0.0 |
| API cannot validate/export |
POST /validate, GET /report.sarif
|
route import smoke test |
| Auth not shared browser→HTTP |
AuthSession.set_cookies() + ReActAgent._sync_browser_auth()
|
3 unit tests |
| 3–4 LLM calls per step | analysis/findings opt-in; reason-only by default | 1-call test |
| 10-iteration cap | MAX_AUTO_ITERATIONS=20 |
config + benchmark default |
| Native tool calling |
complete_with_tools for Anthropic + OpenAI |
5 unit tests |
| LLM ignored login form |
BrowserTool.auto_login + password-form probe |
auto-login unit test |
| IDOR/BOLA verification |
tools/exploit.py identity/path/cookie/param actions |
XBEN-021 solved |
| SSTI evidence |
SSTIScanner emits findings + HTTP evidence |
1 unit test |
| Command injection |
cmd_inject echo-marker verifier |
1 unit test |
| Request replay |
replay with query/body/cookie overrides |
1 unit test |
| Tool options crashed tools |
_tool_options() filters options to the tool signature |
DVWA HTTP/nikto succeed |
| Raw output was overwritten | Raw output and AI analysis are both stored | observation test |
| Anthropic calls could hang ~10 min |
timeout=cfg.timeout, max_retries=1
|
timeout unit test |
| Redundant command-generation call | opt-in via PENTESTAI_GENERATE_COMMANDS
|
4 → 3 LLM calls/step |
M3 policy guards (2026-10-21): hard per-tool/LLM timeouts, 4-scanner recon
budget, cycle detection, endpoint/parameter opportunity hints, browser page-text
preview plus credential-like lines, and budget/loop lessons. The image was rebuilt (pentestai:latest) and
two targets were re-run: XBEN-006 dropped from 703.8 s to 246.7 s and XBEN-021
from 246.9 s to 186.8 s. A later XBEN-021 rerun hit a pathological 985 s stall,
so the container-side timeout -k wrapper remains required.
After preserving the evidence field from tool findings, XBEN-021 was solved
with no probes: 171.5 s, 20 iterations, 5 validated, 0 candidates. This is a
targeted re-run, not a new 20-target rate.
scripts/bench_build.py fixes EOL Debian apt sources (verified on
XBEN-041/091/026), and install.sh now defaults to Docker-only. Details:
MILESTONES.md.
Clean 20-target run (2026-10-24): 8/20 solved (40%), 20/20 runnable,
0 build/up/harness errors, 36 validated, 0 candidates, agent p95 300.9 s and max
301.1 s. Solved: 021, 026, 037, 038, 041, 075, 080, 099. XBEN-021 and 041 are
restored in the same protocol. The host killed the runner at target 17; targets
17–20 were resumed immediately with the same manifest, image, budget, and
probes-off environment (one record per target, no duplicate runs). Artifacts:
artifacts/bench/clean-2026-10-24/.
Still open:
- 5/5 solved on the selected pilot — 021 IDOR, 009 SSTI, 037 blind cmd/OOB, 005 JWT forgery, 099 ObjectId IDOR. Not the full 104 suite; see caveats in BENCHMARK.md.
-
20-target sample: 8/20 — 021/041 restored, but XBEN-018's bundled XSS checker is broken (QtWebEngine
TypeErroron benign input) and five targets produced validated evidence without a flag (006, 008, 002, 101, 087). - ReAct has not autonomously completed an objective on an authorized real-world target — it solves benchmark targets (8/20 no probes), while the authorized viding.org run remained read-only. See MILESTONES.md.
Live verification in the rebuilt container (2026-10-21): Chromium opened
https://example.com (HTTP 200, title captured), screenshot written, and a
demo session produced HTML with VALIDATED/CANDIDATE badges, a SARIF file, and
a 57 KB PDF. The environment smoke test remains 38/53 (15 empty keys).
Real-target read-only run (2026-10-21, authorised scope viding.org): login
succeeded (/login → /v2/beranda), screenshot captured, 2 validated
findings (missing CSP, missing Referrer-Policy), SARIF + PDF generated.
Artifacts: artifacts/real-target/. No scanning, exploitation, or writes.
15 providers registered: Anthropic, OpenAI, Groq, Google, Mistral, xAI, DeepSeek, Together, Perplexity, Azure, Cohere, OpenRouter, Ollama, 9router, Custom.
build_router_from_env() auto-detects from env vars. Streaming via __stream_text__() (anthropic 1.15.0 compatible).
Anthropic was live-verified end-to-end on 2026-10-20 and Ollama (qwen2.5:3b) on 2026-10-24. The 11 cloud providers with API-key env vars have no keys filled in yet; 9router is not running and no custom endpoint is configured.
Validation steps:
# Checks every configured provider, or just one
python3 scripts/live_llm_check.py
python3 scripts/live_llm_check.py --provider ollamaThe loop structure, state machine, 4 system prompts (REACT/PARSING/GENERATION/SUMMARIZATION), and dead-end detection are unit-tested. The loop has completed benchmark objectives (XBEN-021, 041, 037, 080, 038, 099, 075, 026), but not an autonomous objective on an authorized real-world target.
Remaining gaps before a live test:
- Metasploit binary must be installed:
sudo apt install metasploit-framework - HTB VPN must be connected:
openvpn lab.ovpn
Fastest validation target: HTB Jerry (Tomcat WAR upload, no MSF needed) or local DVWA.
get_recommendations() always returns []. Heuristic cross-session learning creates false confidence. The proper replacement is fine-tuning with a real dataset (Phase 6). The module remains to avoid import errors.
| Gap | Status |
|---|---|
| Metasploit binary | ⚠ Not installed — sudo apt install metasploit-framework
|
| HTB VPN | ⚠ Needs to be connected — openvpn lab.ovpn
|
| anthropic SDK streaming | ✅ Fixed — __stream_text__() with fallback |
| LLM providers (15) | ✅ Fixed — auto-detect from env vars |
| LHOST auto-detect | ✅ Fixed — detect_lhost() in msf.py |
| Nmap parser real output | ✅ Verified — list[ParsedHost] correct |
- Assert specific content, not just
is not None - Use the pytest
tmp_pathfixture for file paths - Use a unique
session_idper test to avoid state pollution - Tools that spawn subprocesses: mock
subprocess.run, do not run real binaries in unit tests - LLM calls: always mock
router.complete()except in tests marked@pytest.mark.live