Release v2.6.0 — Parallel crawling and reliable incremental mirrors · PKHarsimran/website-downloader

Faster crawls, safer incremental mirrors, and a sharper README.

🚀 Highlights

Parallel page fetching — new --page-threads option crawls HTML pages concurrently. Benchmarked ~3–4× faster on multi-page sites with 4 workers. The default stays at 1, so crawls remain polite unless you opt in.
Faster parsing — new pip install -e ".[fast]" extra installs lxml, which is auto-detected and used for HTML parsing when available.
--update mode actually works for re-crawls now — see the fixes below; recurring mirrors previously wasted their page budget on bogus 404s.

--update re-crawls no longer discover links from saved pages (whose links are already rewritten to local paths, producing bogus 404 URLs); known pages and assets are re-seeded from the update cache instead.
A 304 Not Modified response with a missing local copy now triggers a clean refetch instead of writing a zero-byte asset or skipping the page.
Assets already present on disk are reported as cached instead of counted as errors when re-running into the same folder.
CSS @import url(...) rules are mapped once; previously the rewritten local path was re-mapped and a garbage download URL was enqueued.
The Rich progress dashboard's cached counter now tracks cached pages and assets correctly.

Pages are parsed from raw bytes so BeautifulSoup honors <meta charset> declarations and BOMs when servers omit a charset header.
CrawlStats gained an errors counter, and the end-of-crawl summary now reports pages, assets, cache hits, and errors.
A page that fails mid-processing is logged and counted as an error instead of aborting the whole crawl.
Removed redundant URL re-parsing in the link discovery hot path; queued assets no longer pre-create directories (failed downloads no longer leave empty folders).
--render-js always uses a single page worker (Playwright's sync API is single-threaded); --delay warns that it applies per page worker.
README restructured with a comparison table (wget / HTTrack / Scrapy), terminal demo, and cookbook layout.

Test suite grew from 23 to 30 tests; every bug fix landed with a regression test that failed before the fix.
Tests pass with and without lxml installed (both parser paths).

Full changelog: v2.5.0...v2.6.0