Skip to content

v1.4.0

Latest

Choose a tag to compare

@leinelissen leinelissen released this 23 Jul 12:42
60f6d95

This release replaces the serial two-phase crawl-then-scrape design with a single
merged task group. Crawling and scraping now run concurrently: each page is
scraped the instant it finishes crawling, without waiting for the rest of the
site.

What changed

  • crawlAndScrape is now a single pipeline. Previously the method crawled
    every page first, collecting all HTML bodies into an array, and only then
    started scraping. Now crawl and scrape tasks share one ThrowingTaskGroup
    as soon as a page is crawled, it's immediately enqueued for parsing, metadata
    extraction, and markdown conversion.
  • Lower memory. HTML bodies are held only until their scrape completes,
    rather than accumulating every page before any scraping begins.
  • Sooner first result. The first ScrapedPage is yielded as soon as the
    first page finishes crawling AND scraping, not after the entire crawl finishes.
  • Bounded scrape concurrency. The number of concurrent scrape tasks is
    capped at CrawlConfiguration.concurrency (default 8), preventing CPU-bound
    parse work from piling up when crawling outpaces scraping.