[Feature Request]: Lightweight HTTP-only crawling mode via DOM AST pruning (no browser required)

[dong7812]

What needs to be done?

Add an optional HTTP-only crawling strategy that fetches pages via plain HTTP and prunes the DOM AST directly — without launching a Playwright browser instance.

The strategy would:

1. Fetch raw HTML via aiohttp/httpx
2. Parse the DOM tree and strip non-content nodes (nav, header, footer, script, style, aside, ads) using a semantic tag allowlist
3. Apply BM25 section filtering (reusing the existing rank_bm25 infrastructure in content_filter_strategy.py)
4. Return clean Markdown — same output contract as today

New class: AsyncHTTPCrawlerStrategy as a sibling to the existing AsyncPlaywrightCrawlerStrategy

What problem does this solve?

For static and SSR-rendered pages (~60–70% of real-world crawl targets), spinning up a Playwright browser adds substantial overhead:

• 300–500ms cold start per page
• 200MB+ memory per browser process

There's currently no way to opt out of the browser layer even when JavaScript rendering isn't needed. This blocks Crawl4AI from being used in high-throughput, resource-constrained pipelines (serverless, edge, batch RAG ingestion).

Target users/beneficiaries

• RAG pipeline developers doing large-scale document ingestion
• Serverless / edge deployments where Playwright is too heavy to run
• Developers who only need to crawl static sites, documentation, blogs, or news
• MCP server authors embedding Crawl4AI as a lightweight tool

Current alternatives/workarounds

• Use requests + BeautifulSoup manually — loses Crawl4AI's BM25, output
  normalization, and async dispatcher
• Use AsyncPlaywrightCrawlerStrategy with headless=True and no JS — still pays
  the full browser startup cost
• Use a separate tool (e.g., dompruner-mcp) outside Crawl4AI's ecosystem

Benchmark from a reference implementation using the same DOM AST approach:

Page	Raw HTML	After pruning	Reduction
docs.python.org	44,316 tokens	1,328 tokens	97.0%
Average (50 sites)	baseline	—	93.5%

Parse time: ~11ms/page vs ~300ms Playwright cold start.
Reference: https://github.com/dong7812/dompruner-mcp

Proposed approach

1. New strategy class AsyncHTTPCrawlerStrategy in async_crawler_strategy.py

   • aiohttp for async HTTP (already a transitive dep)
   • lxml or html.parser for DOM tree traversal (no new deps)
   • Prune by tag allowlist identical to RelevantContentFilter.included_tags

2. Plug into the existing pipeline unchanged:

   • content_filter_strategy.py BM25 → no changes needed
   • markdown_generation_strategy.py → no changes needed
   • AsyncWebCrawler accepts strategy= parameter already

3. Opt-in via config:
   crawler_config = CrawlerRunConfig(crawler_strategy="http")

Foreseeable challenges:

• Pages that rely on client-side rendering will return empty/partial content
  → document clearly; auto-fallback to Playwright could be a follow-up
• robots.txt handling needs to be preserved (currently in Playwright strategy)

Success metric: same Markdown output quality as Playwright for static pages, with ≥10x reduction in memory footprint and ≥5x reduction in per-page latency.

Happy to prototype a PR targeting the develop branch.

[IlyaKhalafi]

The browser-free transport already exists as AsyncHTTPCrawlerStrategy. Pass its instance to AsyncWebCrawler(crawler_strategy=...), rather than adding crawler_strategy="http" to CrawlerRunConfig.

That keeps the existing scraping and Markdown pipeline. Configure excluded_tags and DefaultMarkdownGenerator(content_filter=BM25ContentFilter(...)) on the run config. Robots checking belongs to AsyncWebCrawler.arun and is enabled with check_robots_txt=True.

I ran this path on 0.9.5 from that revision against a local HTTP fixture. It produced raw Markdown, removed the excluded navigation, retained the relevant paragraphs in BM25 fit Markdown, and blocked a disallowed URL before fetching that page. No browser or external website was used.

The useful next comparison would be your pruning approach against this existing HTTP path, with the same pages and reused sessions. Check preservation of task-relevant tables, code and links alongside token count and latency. A large token reduction alone does not establish equivalent extraction quality, and the proposed speed and memory gains still need measurement.