This release replaces the serial two-phase crawl-then-scrape design with a single
merged task group. Crawling and scraping now run concurrently: each page is
scraped the instant it finishes crawling, without waiting for the rest of the
site.
What changed
crawlAndScrapeis now a single pipeline. Previously the method crawled
every page first, collecting all HTML bodies into an array, and only then
started scraping. Now crawl and scrape tasks share oneThrowingTaskGroup—
as soon as a page is crawled, it's immediately enqueued for parsing, metadata
extraction, and markdown conversion.- Lower memory. HTML bodies are held only until their scrape completes,
rather than accumulating every page before any scraping begins. - Sooner first result. The first
ScrapedPageis yielded as soon as the
first page finishes crawling AND scraping, not after the entire crawl finishes. - Bounded scrape concurrency. The number of concurrent scrape tasks is
capped atCrawlConfiguration.concurrency(default 8), preventing CPU-bound
parse work from piling up when crawling outpaces scraping.