Skip to content

v2.0

Choose a tag to compare

View all tags
@PKHarsimran PKHarsimran released this 19 Jul 12:29
v2.0
ab31fc3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

📦 Website Downloader v2.0 — Release Notes

Tag: v2.0 • Date: 2025-07-19

✨ Highlights

Area What’s new
Command-line UX • All interactive input() prompts replaced with argparse flags.
 --url, --destination, --max-pages, --threads (+ rich --help).
Logging • Clean, colour-friendly log format (%(asctime)s | %(levelname)s | …).
• Fixed %-format bug that broke Python 3.9/3.10 runners.
Crawler core • Correct handling of pretty URLs (/about/)—no more unwanted .html suffixes.
• Eliminated “double-domain” save paths.
• Thread-pool asset fetcher (default 6 workers).
Reliability • requests session with auto-retry (5×, exponential back-off).
• 15 s timeout & path-sanitisation against ..\ / \ tricks.
Performance • Breadth-first crawl queue + concurrent downloads.
• Crawl summary shows total time & average latency.
CI / Packaging • GitHub Actions smoke-test mirrors 5 pages on Python 3.9 → 3.12.
• requirements.txt trimmed to requests, beautifulsoup4, urllib3.

🚀 Quick-start

python website_downloader.py \
       --url "https://example.com" \
       --destination "archive" \
       --max-pages 50 \
       --threads 8

If --destination is omitted the script defaults to example_com/.

🔄 Migration notes

  • Rename any scripts that relied on interactive prompts—now supply flags.
  • Logs are written to web_scraper.log (same folder as the script).
  • Default --max-pages lowered 100 → 50; bump it as needed.

Happy mirroring! 🎉

Assets 2
Loading