Skip to content

v0.3.0

Latest

Choose a tag to compare

@2913636 2913636 released this 06 Jul 14:18

Async Support

  • extract_async() on ExtractionStage — default wraps sync in executor, zero breaking changes
  • Pipeline.run_async() fully async: parallel search, verify, AND extraction phases
  • Crawler.crawl_stream() truly parallel via asyncio.Semaphore
  • CrawlConfig.max_concurrent for configurable concurrency

Browser Pool

  • BrowserPool class: pre-warm and reuse StealthySession instances
  • Thread-safe sync + async APIs, idle timeout eviction, stats monitoring
  • BrowserPoolExhausted for clear error handling at capacity

Concurrency Config

  • PipelineConfig.enable_concurrent_extraction — parallelize across candidates
  • PipelineConfig.max_concurrent_candidates

Integration Tests

  • tests/conftest.py with pytest markers (integration, network, slow) + flag gating
  • tests/integration/ — real-URL end-to-end pipeline tests
  • 275 → 284+ tests

Fixed

  • CurlCffiStage.can_handle() string-type crash
  • crawl_stream(): time.sleep() → asyncio.sleep()