PhantomSpider is a stealth-ready, high-performance Python web crawler built on Breadth-First Search (BFS) for structured and systematic URL discovery.
Crafted for penetration testers, bug bounty hunters, and digital forensics professionals, it combines speed, scope control, and stealth features like random user-agent rotation to evade detection.
-
🔍 Breadth-First Search (BFS) crawling for wide and systematic discovery
-
🎨 Rich terminal UI with color-coded tables and progress indicators
-
🌐 Domain-restricted crawling to keep exploration in-scope
-
🧠 Parameter-based deduplication (
--dedup-params) for clean results -
⚡ Random User-Agent rotation to bypass simple bot detections
-
⏳ Configurable timeouts & delays for crawl tuning
-
📂 Export findings to structured reports:
urls_discovered.txtparameters_found.txtdeduped_params.txt
git clone https://github.com/rey2044/PhantomSpider.git
cd PhantomSpider
python3 -m venv venv
# Windows
venv\Scripts\activate
# macOS / Linux
# source venv/bin/activate
pip install -r requirements.txtpython PhantomSpider.py -u <url> [options]# Crawl a single target
python PhantomSpider.py -u https://example.com
# Crawl from a list with timeout & delay
python PhantomSpider.py -l urls.txt -t 1 --req-timeout 10
# Save crawl results
python PhantomSpider.py -u https://example.com -s
# Deduplicate query parameter patterns
python PhantomSpider.py -u https://example.com --dedup-params
# Full stealth mode (save + dedup + random user agents)
python PhantomSpider.py -u https://example.com -s --dedup-params --rua| File | Description |
|---|---|
urls_discovered.txt |
All discovered and visited URLs |
parameters_found.txt |
URLs containing query parameters |
deduped_params.txt |
Unique entries per parameter key structure |
- Starts with a seed URL or a list of targets.
- Traverses links using BFS crawling.
- Restricts exploration to in-scope domains only.
- Detects and deduplicates parameterized URLs.
- Outputs results with a color-coded, professional terminal display.
- Python 3.8+
requirements.txt:
requests
beautifulsoup4
rich
playwright
⚠️ After installing Playwright, runpython -m playwright installto download browser binaries.
Released under the MIT License – see LICENSE.
Laykumar Patel Cyber Security Researcher | Digital Forensics & VAPT Expert
📧 Email: laykumarhp67428@gmail.com 📞 Contact: +91 9327112912 🔗 GitHub: @rey2044
Feedback and contributions are welcome! 👉 Open an Issue or submit a Pull Request.