Version 2.0

🚀 New Features & Enhancements

🔥 Performance & Scalability Improvements

• Fully asynchronous architecture using aiohttp for non-blocking HTTP requests.
• Parallel sitemap parsing for faster URL discovery and processing.
• Connection pooling & request batching for optimal network efficiency.

🎯 Improved Web Scraping & Phrase Matching

• Recursive sitemap discovery to identify hidden URLs.
• Optimized HTML parsing via BeautifulSoup for accurate text extraction.
• Regex-based phrase matching (optional) for flexible scanning.
• Case-insensitive search for more robust results.

🛡️ Security & Error Handling

• Robust exception handling with structured logging and retry logic.
• Automatic exponential backoff for transient network failures.
• Defused XML parsing for enhanced security.

📊 Better Data Output & Storage Options

• Support for multiple output formats: TXT, JSON, CSV.
• Efficient file I/O operations to minimize disk usage.
• Real-time progress tracking via tqdm.

🛠️ Configurability & User Experience

• Command-line options for flexible scanning (--regex, --concurrency, --output, --timeout).
• Automatic domain protocol detection (HTTPS preferred, fallback to HTTP).
• Verbose logging mode (--debug) for better troubleshooting.

📝 Known Issues

• Some highly dynamic sites may block rapid requests; consider rate limiting.
• Regex-based phrase search may impact performance on large pages.