Repository navigation
Roadmap 12M
Window: November 2026 → October 2027
Baseline: 24 October 2026
Goal: a model-agnostic, self-hosted, evidence-first pentest platform that can run
any LLM/agent behind the same control plane, with Strix as one swappable engine
rather than a permanent dependency.
Any model, any agent, same guardrails. Results may vary by model strength, but scope, approval, validation, evidence, and reporting never do.
Success is measured on four axes, not one:
| Axis | Metric | Target |
|---|---|---|
| Capability | Solved targets on the fixed 20-target sample | ≥18/20 |
| Model-agnostic | Top-3 models within 15% on the same harness | 3 models |
| Trust | Validated precision / scope violations | 100% / 0 |
| Team readiness | Workspaces + SSO + quotas | 1.0 |
The benchmark is a regression signal, not a market claim. XBOW-104 is saturated and deprecated; every external claim must include method, model, budget, and harness version.
| Area | Current state |
|---|---|
| Test suite | 2073 collected · 2053 passed · 20 skipped |
| Benchmark | 8/20 (40%), 0 build/up/harness errors, 018 checker environment-blocked |
| XSS | Browser-execution verifier done; DVWA confirmed |
| Auth | Registration + auto-login fixed; 024 SSRF chain verified manually |
| Engines | Native ReAct only; no Strix/Shannon adapter |
| Models | 15 providers; Anthropic + Ollama live-verified; 11 cloud keys empty |
| Evaluation | 20-target manifest + scorecard; no model matrix |
| Platform | 48 API endpoints, RBAC, audit, evidence, PTES/ISO/SARIF/PDF |
| Teams | Single-tenant; roles and session sharing exist |
| UX | CLI/API; TUI done (textual 5-tab dashboard) |
| Setup | Auto-setup script; modular deps (core 12 / full 38); scan presets; guided wizard |
| Ops | Docker Compose; disk-constrained (≈8–15 GB free) |
| # | Workstream | Owner outcome |
|---|---|---|
| W1 | Agent runtime |
AgentBackend contract; ReAct, Strix, CLI-agent adapters |
| W2 | Model layer | Capability profiles, JSON repair, routing, fallback, local models |
| W3 | Tools & verifiers | XSS/SSRF/auth/business-logic breadth; evidence-first |
| W4 | Evaluation | Model matrix, fixed manifests, trace schema, reproducibility |
| W5 | Control plane | Scope, brief, audit, validation, evidence, reports |
| W6 | Teams & enterprise | Workspaces, RBAC, SSO, quotas, retention |
| W7 | UX | CLI → TUI → optional web client after API freeze |
| W8 | Ops & security | Sidecar isolation, disk, backups, observability |
| W9 | Distribution & setup | PyPI publish, auto-setup, modular deps, one-command install |
Theme: make the harness work with any model, and close the cheapest gaps.
| Month | Deliverable | Exit signal |
|---|---|---|
| M1 (Nov) |
AgentBackend interface; ReAct as backend #1; --model + model_id in benchmark records |
Two models run the same manifest with recorded metadata |
| M1 | 024 re-run with adaptive step timeout; 091 auth run; near-miss flag extraction | 10–12/20 |
| M2 (Dec) | Model capability profiles; JSON repair/retry; per-family prompts; adaptive budget | Weak local model completes a run without crashing |
| M2 |
Strix OSS adapter MVP via isolated sidecar (strix -n + run artifacts) |
One Strix scan imports into a PentestAI session |
| M2 | Strix adapter delivered early (CLI + artifacts; live DVWA import) | 1 XSS finding imported as candidate |
| M3 (Jan) | Model matrix (Claude, DeepSeek, Qwen/Ollama; add keys where possible) | First public-internal leaderboard |
| M3 | Live lab: DVWA + Juice Shop + Pentest-Ground :81
|
≥2 classes validated outside XBOW |
| M3 | PyPI publish (pip install pentestai); Homebrew tap; one-liner install | pip install pentestai && pentestai guided works from scratch |
| M3 | Scan profiles: compliance (PCI-DSS, SOC2), bug-bounty, red-team | 3 additional presets beyond quick/owasp10/api/full |
Gate G1: 3 models on one harness; ≥12/20; Strix import works; live lab has at least one validated XSS/SSRF finding; PyPI package published.
Theme: stop being a wrapper; own the parts that matter.
| Month | Deliverable | Exit signal |
|---|---|---|
| M4 (Feb) | Strix adapter production: jobs, artifacts, finding mapping, re-validation, cost/trace | Strix findings enter the validation gate, not directly the report |
| M4 | Trace schema + playbook extraction into strategy memory | Reusable lessons, no target-specific hints |
| M5 (Mar) | Own engine replaces Strix for 3 classes (XSS, SSRF, auth) | Own engine ≥70% of Strix solve set on those classes |
| M5 | Shannon adapter evaluation only (AGPL/internal, source-aware) | Go/no-go note; no default dependency |
| M6 (Apr) | Fleet/ensemble: 2 models on hard targets, merge validated findings | Measurable lift without false positives |
| M6 | TUI MVP after API/AgentBackend freeze (3–4 weeks) | Session, plan stream, findings, evidence, approval, stop |
Gate G2: ≥16/20; own engine ≥70% of Strix; TUI usable by one operator; 5 vulnerability classes validated in the live lab.
Theme: make it safe for more than one team.
| Month | Deliverable | Exit signal |
|---|---|---|
| M7 (May) | Workspaces/tenant-scoped sessions, findings, evidence, audit; team RBAC | Two teams isolated in one instance |
| M7 | SSO/OIDC, quotas per workspace, LLM budget per team | Admin can onboard a team without shell access |
| M8 (Jun) | Retention/deletion policies, audit export, white-label reports | Client data removable; report can carry firm branding |
| M8 | Cost/latency: model routing, caching, p95 budgets | Cost per solve ≤ Q1 baseline |
| M9 (Jul) | Three consecutive clean runs; no regressions; independent scorecard | ≥18/20 stable |
| M9 | TUI polish or minimal web dashboard (choose one) | Operators stop needing raw API |
Gate G3: 18/20 across 3 runs; team isolation proven; SSO; cost and p95 within budget.
Theme: Strix becomes optional, not structural.
| Month | Deliverable | Exit signal |
|---|---|---|
| M10 (Aug) | Own engine default for standard web/API; Strix fallback only | ≥90% of Strix solve set without Strix |
| M10 | Source-aware mode decision: Shannon commercial license or own SAST | Documented decision |
| M11 (Sep) | Enterprise readiness: SSO/SCIM, audit export, HA/backup, security review | Internal security sign-off |
| M11 | External validation on authorized lab targets | No scope violations; evidence reproducible |
| M12 (Oct) | Release 1.0; publish methodology and limitations | Independent report, not a vendor claim |
Gate G4: Strix optional; 1.0 shipped; live validation published with caveats; no critical security findings in the control plane.
| Level | Meaning | Target date |
|---|---|---|
| L1 Legal/operational | Fork/apache code; no account or upstream API | M1 |
| L2 Functional | Own engine runs the same targets without Strix | M6 |
| L3 Quality | ≥90% of Strix solve set on our manifest | M10 |
| L4 Production | Live lab/authorized target, 3 clean runs, cost parity | M12+ |
Strix stays as an optional adapter even after L4. The objective is swappability, not deleting a good engine.
| Scenario | 12-month outcome |
|---|---|
| 1 senior + AI | G1–G2 likely; G3 slips 2–4 months; TUI/web late |
| 2 engineers + AI (recommended) | G1–G4 on schedule; TUI parallel after API freeze |
| 3 engineers | 30–40% faster; add live validation and enterprise |
| 5 engineers | 6–8 months to G3; only if commercial demand is proven |
Critical path: AgentBackend → model profiles → Strix adapter → own engine → stable benchmark. TUI and multi-tenancy are parallel, not blocking.
| Risk | Mitigation |
|---|---|
| Benchmark overfitting | Fixed seed + held-out live labs; no target-specific lessons |
| Saturated XBOW suite | Treat as regression, not market proof; publish method |
| Model latency/timeouts | Capability profiles; adaptive step timeout; fallback chain |
| Provider ToS / fine-tuning | No distillation of hosted outputs into competitors; open-weight only |
| License contamination | Strix Apache-2.0; Shannon AGPL only internal/white-box or commercial license |
| Docker-in-Docker | Isolated runner/sidecar; never mount the socket into the app container |
| Disk exhaustion | Per-target image cleanup; benchmark only on a clean host |
| Data leakage across teams | Workspace scoping before multi-team release; retention policy |
| Scope bypass | PermissionGate outside the model; audit every action |
| TUI/feature creep | No TUI before API freeze; strict 3–4 week MVP |
- Do not build a TUI before
AgentBackend/API is stable. - Do not build multi-tenancy before two real teams need it.
- Do not fine-tune a model to replace reasoning you do not have yet.
- Do not claim Strix-level scores before a same-manifest, same-model run.
- Do not let an external engine hold scope, approval, or “validated” status.
- Do not chase the 104-target suite at the expense of live lab validation.
-
AgentBackendcontract + ReAct backend. -
--modelflag andmodel_id/harness_versionin benchmark records. - JSON repair + retry + schema validation.
- Re-run 024, 091, and near-miss targets (006, 008, 002, 101, 087).
- Strix OSS sidecar adapter spike.
- Canonical trace schema.
- Live lab runner (DVWA + Juice Shop + Pentest-Ground).
- Model matrix v1 (Anthropic + Ollama + one cloud key).
- 18/20 fixed sample, three consecutive runs, 0 harness errors.
- At least 5 vulnerability classes validated in the live lab.
- Two models within 15% on the same harness, metadata recorded.
- Strix optional; own engine handles standard web/API.
- Workspaces + SSO + quotas + retention.
- Zero scope violations in the last 90 days.
- No critical findings in the control plane.
- Documentation states limitations as clearly as capabilities.
This roadmap is a plan, not a promise. Dates move when the evidence moves.