FInal repo#48
Open
gurunathan-r wants to merge 28 commits intohacktofuture-sjec:mainfrom
Open
Conversation
Thin FastMCP passthrough over Kali binaries (nmap, httpx, katana, nuclei, gobuster, arjun, x8, rustscan, masscan, smbmap, etc.) plus three parallel fan-out workflows: web_reconnaissance, api_testing, network_discovery. Every tool call spawns an independent subprocess with no semaphore gating; long scans return a job_id immediately via an in-memory job registry so the MCP SSE session never blocks. Results are raw parsed dicts — no summarization — the agent owns decision-making. Deploys to /opt/kali-mcp on the Kali VM via install.sh + systemd unit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Neutral, vendor-agnostic name that scales as more red-team tooling gets layered on (not Kali-specific). Updates the Python package, env vars (RED_ARSENAL_*), install dir (/opt/red-arsenal), log dir, systemd unit, and FastMCP service name. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Go compilation of httpx/nuclei/etc was OOM-killing Kali VMs with <3GB RAM. Switched to downloading precompiled release binaries from GitHub — drops install time from ~15min to ~30s and uses essentially no RAM. Adds --skip-binaries and --skip-apt flags for faster iteration on the Python server without re-running tool downloads. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Four issues surfaced from the first end-to-end test against a real Kali VM: 1. Kali ships `python3-httpx` at /usr/bin/httpx — a completely different tool from ProjectDiscovery httpx. Force-install the ProjectDiscovery binary to /usr/local/bin and reorder the systemd PATH so it wins. 2. Go tools (katana) need HOME to resolve ~/.config. The systemd unit ran with no HOME set, so katana crashed with "$HOME is not defined". Added Environment=HOME=/root. 3. pipx-installed tools (arjun, dirsearch, paramspider) landed in the invoking user's ~/.local/bin, invisible to the root-run systemd service. Switched to `sudo pipx install` with PIPX_BIN_DIR=/usr/local/bin so shims are globally visible. 4. ffuf's `-o -` stdout redirection is unreliable across versions. Use a temp file like arjun/paramspider and read it back into RunResult. Also tightens the smoke test's nuclei call (severity=critical) so it finishes in under a minute instead of running 2000+ info-level templates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…n/x8 Second round of fixes from the live Kali run: - ToolSpec.resolve() now checks extra_paths FIRST and only falls back to shutil.which. This is critical for httpx: the venv pip-installs the Python httpx CLI as a transitive dep of fastmcp, and it was winning over our ProjectDiscovery binary. Pin httpx/katana/gau/waybackurls/nuclei/x8/ rustscan to their canonical /usr/local/bin paths. - paramspider isn't on PyPI — switch pipx to git+https URL. - rustscan is in Kali's apt repo; drop the flaky GitHub fetch block and let apt handle it. - x8 asset pattern corrected to match Sh1Yo's actual release naming (`x8-<target>.tar.gz`). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…b fetch Kali's default apt repo does not package rustscan. Reverting to a best-effort .deb fetch from RustScan/RustScan GitHub releases. Failures are silent and non-fatal since nmap+masscan cover port scanning. Also adds git to the apt list (needed for pipx git+https paramspider install). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eight PASSes out of fourteen runnable tools on the first full run. The
remaining failures were all wrapper/parser/config bugs — not MCP or
concurrency issues. This commit addresses them:
- parsers.py: katana JSONL uses `request.endpoint` / `request.url`, not
top-level keys. Fixed the parser to walk the nested structure and
also surface method + status_code.
- config.py: default wordlists were too large for anything interactive.
Point gobuster + ffuf at /usr/share/wordlists/dirb/common.txt (~4.6K
entries).
- install.sh:
* pipx inject dirsearch setuptools — Python 3.12+ stopped bundling
setuptools in new venvs, and dirsearch still imports pkg_resources.
* Add `wordlists` apt package so /usr/share/wordlists/dirb is present.
- tools/api.py paramspider_impl: git-installed paramspider only accepts
`-d DOMAIN` and writes to `results/<domain>.txt` in cwd. Run it in a
temp dir and read the file back.
- tools/network.py:
* _resolve() helper for hostname→IP (masscan rejects hostnames).
* masscan_impl now pre-resolves and uses a narrower default port list
(common service ports) so smoke runs don't take 10 minutes.
* nmap_advanced_impl: os_detection now defaults False (was adding
1–3 min per host with -O).
* smbmap_impl: dropped `-R --depth 2` (smbmap 1.10 changed -R to take
a share-name arg, not a flag).
The MCP server plumbing itself — parallel job spawning, SSE transport,
job status/result/cancel polling, server_stats, parser registration,
the installed-tools introspection — all worked flawlessly on the first
run. These are just tool-CLI drift fixes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
pipx inject was silently failing on the first Kali run — setuptools
didn't end up in /opt/pipx/venvs/dirsearch so dirsearch still crashed
at import with ModuleNotFoundError: pkg_resources.
Fall back to calling the venv's pip directly:
/opt/pipx/venvs/dirsearch/bin/pip install setuptools
Round-2 smoke test otherwise went from 8 → 12 passes out of 20:
* paramspider PASS (git-CLI cwd results fix)
* ffuf PASS (wordlist path fix)
* masscan PASS (hostname resolver)
* smbmap PASS (simplified args)
Only dirsearch remained red among actionable bugs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First backend→arsenal connection. The three recon endpoints now invoke
real Kali tools through the MCP SSE transport and return parsed findings:
- POST /scan/network → run_nmap (configurable ports, defaults to a
common-service sweep).
- POST /scan/web → web_reconnaissance workflow (wait=True) with an
1800s poll ceiling for the slower fan-out members.
- POST /scan/system → run_smbmap (SMB tier is enough for v1).
Each scan also pushes a `{type: "tool_call"}` envelope over /ws/red at
submit + finish so the frontend tool-call stream mirrors the MCP job
lifecycle in real time. Broadcast is best-effort and can never break
a scan.
New module:
- red_agent/backend/services/mcp_client.py — opens a fresh Client() per
call, submits via call_tool, polls job_status, fetches job_result.
No long-lived SSE session across FastAPI request boundaries.
Config:
- MCP_SERVER_URL env var (default http://127.0.0.1:8765/sse) — override
when the arsenal runs on a different host.
- fastmcp>=0.4.0 pinned in root requirements.txt.
Out of scope for this step: run_cloud_scan, lookup_cve, run_exploit,
run_cve_exploit, plan_attack, evolve_strategy — all remain mocked and
will be wired once the exploit/CVE-feed MCP surface is built.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tion
calling
- Recon Agent: Groq SDK function calling, LLM decides tools (nmap, nuclei, gobuster, ffuf,
sqlmap)
- Exploit Agent: LLM-driven SQLi/LFI/CmdInjection exploitation with deterministic fallback
- Auto-trigger: recon.complete → EventBus → exploit starts autonomously
- CVE Fetcher: NVD API integration
- Report: downloadable pentest report at /report/download/{id}
- EventBus: async pub/sub connecting both agents
- 6 passing tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Dark violet themed frontend (Orbitron + JetBrains Mono fonts) - Chat interface powered by NVIDIA Qwen/Llama LLM - Mission orchestrator: parallel recon → LLM analyze → plan → exploit → report - 6 recon tools run in parallel via asyncio.gather (nmap, httpx, nuclei, dirsearch, katana, gobuster) - Real-time WebSocket streaming of tool calls, logs, mission phases - Mission control: pause/resume/abort from frontend - LLM generates security analysis, attack plans, and pentest reports - Target parsing for URLs, IPs, and ports Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Resolved conflict in recon_agent.py — kept Prathiba's full Groq function-calling agent implementation over the stub. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Switched ReconAgent and ExploitAgent from Groq SDK to NVIDIA NIM - Unified llm_client.py with chat(), chat_json(), tool_call() functions - Orchestrator delegates to Prathiba's agents (ReconAgent + ExploitAgent) - Fixed tool availability for remote MCP (no local install check) - Sync LLM calls in chat endpoint (3s response vs 30s+ timeout) - Dark violet theme, fixed chat ordering Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Switched to CrewAI with 3 agents (Recon, Analyst, Exploit) - All agents use NVIDIA NIM Llama 3.1 70B via litellm - CrewAI Tool wrappers broadcast each tool call to dashboard via WebSocket - Tool cards show actual findings (ports, services, vulns, URLs) - Agent name displayed on each tool card - CLEAR buttons on Activity Panel and Live Log - Improved chat system prompt for natural conversation - Fixed uvicorn event loop with --loop asyncio Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Merged Red Agent dashboard (CrewAI + Groq agents, rich UI) - Unified event_bus.py: Blue uses emit()+worker, Red uses publish()+history - Restored main.py to run both backends (Red:8001, Blue:8002) - Merged requirements.txt (paramiko + crewai + openai + fastmcp) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Author
|
d |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.