Skip to content

Releases: gapilongo/pentest-copilot

Release list

v13.7 — first public release

v13.7 — first public release Latest
Latest

Choose a tag to compare

View all tags
@gapilongo gapilongo released this 27 May 15:52
v13.7
6f55a76

v13.7 — first public release

Pentest Copilot's first cut as a public, documented, license-stamped project. The system is operator-substrate–first (38 catalog entries, 16 playbooks, 6 RAG packs, 4 deterministic tools), with structural verifier rules that physically block known regression patterns at composition time, an LLM-as-judge quality eval framework calibrated to 97%, and a graceful failure UX that never returns "system rejected."

Runs on one box (default: 4×A10 / 92 GB VRAM, Qwen3-Next-80B-A3B-FP8 backend). No external API. Authorized engagements only.

Highlights

  • Substrate-first architecture — answers come from hand-curated technique catalog entries, multi-phase playbooks, freshness-aware RAG packs, and deterministic tool calls. Model recall is the last resort, not the first.
  • 17 structural verifier rules — every answer runs through them before shipping; hard failures trigger regen with a constraint message. Bugs that get fixed stay fixed (the v3/v4 "indistinguishable from admin" / PwnKit-as-kernel / phase-numbering-skip patterns are now structurally impossible to ship).
  • 97%-calibrated LLM-as-judge quality eval framework (eval/quality/judge.py) with 5 hand-written reference answers as oracle. Current baseline: 27/30 (8.7/10) on the judged 5-case set.
  • 151-prompt × 30-topic coverage corpus with substrate-hit 92.7%, 1 hard failure, 0 fake citations across the full corpus.
  • Asset-shape scope filter — kills alarm-fatigue from github.com, os.system, mysql.func, 169.254.169.254 and similar non-target tokens that produce garbage scope warnings on other tools.
  • Knowability routing — 4-class (verifiable_fact / time_decaying_documented / pure_reasoning / unverifiable_prediction) with class-specific stance enforcement. Pure-reasoning prompts get honest-stop refusals when recon evidence is missing; verifiable-fact prompts force deterministic tool calls.
  • Graceful failure UX — when the verifier rejects, the system ships the substance with the offending claims surgically removed and a visible banner, not a "system rejected" notice. When the answer truncates, a ⚠ Answer truncated banner tells the operator how to continue. One bad refusal in a critical moment costs more trust than ten missed playbooks.

What's in this release

  • src/agent/provenance_verifier.py — tag validity + 10 regression patterns + 5 truncation detectors + phase monotonicity
  • src/agent/technique_catalog.py + 38 YAML entries (ESC1-15, Kerberoasting, RBCD, SUID abuse, web SQLi/XSS/SSRF/XXE, GTFOBins primitives, …)
  • src/agent/playbook_matcher.py + 16 playbook YAMLs (low-priv Linux/Win privesc, AD enumeration, ADCS triage, post-DCSync, external recon, web app pentest, …)
  • src/agent/knowability_router.py + LLM-fallback classifier
  • src/agent/rag_loop.py + 6 freshness-aware RAG packs (operator_hacktricks, cve_kev, attack_mitre, nvd_cve, github_releases, research_blogs)
  • src/agent/deterministic_tools.py — lookup_cve, check_kev, lookup_attack_id, fetch_research_blog
  • src/engagement/scope.py + output_filter.py + input_scope_filter.py — asset-shape gate with tool-repo / DB-system / cloud-metadata / config-filename denylists
  • src/agent/answer_rewriter.py — section composition + banners + graceful fail-open + scope strip
  • eval/quality/ — judge module + calibration suite + 5 reference answers + rubric
  • eval/coverage/ — 151-prompt corpus runner + aggregator
  • frontend/ — React 18 + Vite + Tailwind chat UI with AuditPanel surfacing per-turn signals

Quality numbers (verifiable in docs/)

Metric Value Source
Senior-graded quality (5-case oracle) 8.7 / 10 docs/quality_v13_7_*.md
Substrate-hit % (151-prompt coverage corpus) 92.7 % docs/v13_6_qwen36_COVERAGE_MATRIX.md
Hard failures across 151 prompts 1 same
Fake citations across 151 prompts 0 same
Judge calibration pass rate 97 % docs/quality_judge_calibration.md

Architecture

See README.md for the full architecture diagram, request lifecycle sequence diagram, knowability routing decision tree, and verifier + regen loop diagram.

Install + run

Full installation guide is in README.md → Installation. Short version:

export VENV=$HOME/pentest-venv
export MODELS_DIR=$HOME/models

python3.10 -m venv $VENV && source $VENV/bin/activate
pip install vllm transformers torch accelerate
pip install -r requirements.txt
pip install fastapi 'uvicorn[standard]' httpx pyyaml pydantic huggingface_hub

mkdir -p $MODELS_DIR
hf download Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 --local-dir $MODELS_DIR/Qwen3-Next-80B-A3B-Instruct-FP8

# Rewrite launchers to your venv + model paths (one-shot)
find scripts -name 'serve_*.sh' -exec sed -i "s|/home/ubuntu/vllm-env|$VENV|g; s|/home/ubuntu/models|$MODELS_DIR|g" {} +

# Bring it up
./scripts/serve_qwen3_next_80b_fp8.sh                    # vLLM, port 9000
./scripts/serve_case_workspace_qwen3_next.sh             # case_server, port 8000

Adding content

The substrate is the primary lever for quality. To add a technique catalog entry, a playbook, or a RAG pack, see CONTRIBUTING.md for full walkthroughs.

Breaking changes

None — first public release.

What's NOT in this release (honest limitations)

  • Not autonomous. Renders commands for the operator to run; no tool-execution sandbox.
  • Not a senior-replacement. ~8.7/10 quality means roughly 1 in 4 answers needs an operator edit.
  • Cloud / container / k8s substrate is thin vs Linux/Windows/AD/ADCS/web coverage.
  • Judge is the same family as the system being judged (Qwen). Calibrated, but has a known false-positive on PKINIT EKU OID.
  • No real-time RAG pack updates — ingest scripts are manual.
  • AuditPanel surfaces substrate signals but not which verifier rules rejected — data is in the response audit, just not rendered in the UI yet.

Prior release notes (for historical context)

Earlier versions are documented in:

  • RELEASE_NOTES_v13.md — v13 production push (structured catalog, playbook layer, knowability router, tool-mediated facts)
  • RELEASE_NOTES_v13_1.md — content expansion (catalog 6→26, playbooks 1→10)
  • RELEASE_NOTES_v12.md — freshness-aware RAG, MITRE/NVD/SpecterOps ingest, provenance verifier v1
  • RELEASE_NOTES_alpha_v0.1.md — original alpha
  • v13.2 through v13.6 (intermediate iterations) are covered in commit history; v13.7 commit summarizes them

License

Apache 2.0 — see LICENSE. License-aligned with the Qwen model weights this system runs on.

Security disclosure

Vulnerabilities should be reported via GitHub Security Advisories — not as public Issues. See SECURITY.md for the full policy and response timeline.

Authorized use

Designed for authorized engagements only — lab work, CTFs, owned systems, sanctioned pentests. Authorization is the operator's responsibility.

Assets 2
Loading