[Bug]: Not able to use deep_crawl_strategies while using arun_many

[RaviKasaudhan]

crawl4ai version

0.7.4

Expected Behavior

It was supposed to deep crawl all the urls provided while doing arun_many.

Current Behavior

Instead of crawling with deep crawl, it is giving failed to crawl.

Is this reproducible?

Yes

Inputs Causing the Bug

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
from crawl4ai.content_scraping_strategy import LXMLWebScrapingStrategy
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    config = CrawlerRunConfig(
        deep_crawl_strategy=BFSDeepCrawlStrategy(
            max_depth=20,
            max_pages=5000,
            include_external=False,  # stay inside site
        ),
        cache_mode=CacheMode.ENABLED,
        exclude_external_links=True,
        exclude_social_media_links=True,
        stream=True,  # important for streaming results
        page_timeout=240000,
        scraping_strategy=LXMLWebScrapingStrategy(),
        wait_until="domcontentloaded",
        semaphore_count=3,
        markdown_generator=DefaultMarkdownGenerator(content_source="raw_html")
    )

    async with AsyncWebCrawler() as crawler:
        # Step 1: await arun_many to get async iterator
        result_iterator = await crawler.arun_many(
            urls=["https://leclairfoundation.org", "https://jottful.com"],
            config=config
        )

        # Step 2: iterate over results as they complete
        async for result in result_iterator:
            if result.success:
                print(f"Just completed: {result.url}")
                # process_result(result)
            else:
                print(f"Failed: {result.url} – {result.error}")

asyncio.run(main())

Steps to Reproduce

Code snippets

OS

macOS

Python version

3.12

Browser

Chrome

Browser version

No response

Error logs & Screenshots (if applicable)

python3 scrapping_crawl4ai_current_solution.py [INIT].... → Crawl4AI 0.7.4 Failed: https://jottful.com Failed: https://leclairfoundation.org

[Cuiyansong]

same issue & question why?

[Ahmed-Tawfik94]

Root Cause Analysis

Primary Cause:

The arun_many method in crawl4ai/async_webcrawler.py does NOT have special handling for config.deep_crawl_strategy. It simply passes the config to the dispatcher, which processes each URL individually without invoking the deep crawl logic.

Evidence:

1. Code inspection: Searching for deep_crawl_strategy in arun_many returns no matches
2. Historical fix: Commit f7ce2d4 (Jan 30, 2025) added this functionality but it's NOT in the current codebase
3. Test confirms: The test script explicitly checks for the fix and confirms it's missing

Why the decorator alone doesn't work:

While arun is decorated with DeepCrawlDecorator, the problem is:

1. arun_many uses a dispatcher that processes URLs in parallel
2. Each URL is processed as a separate crawl task by the dispatcher
3. The deep crawl strategy is designed to crawl ONE start URL and discover additional pages from it
4. The dispatcher calls crawler.arun() for each URL, but doesn't handle the deep crawl strategy properly

Suggested Fixes (Backward-Compatible)

The fix from commit f7ce2d4 should be applied to crawl4ai/async_webcrawler.py:

1. Add helper function to merge async generators (before arun_many):

async def merge_async_generators(generators):
    tasks = {asyncio.create_task(gen.__anext__()): gen for gen in generators}
    while tasks:
        done, _ = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
        
        for task in done:
            gen = tasks.pop(task)
            try:
                result = task.result()
                yield result
                tasks[asyncio.create_task(gen.__anext__())] = gen
            except StopAsyncIteration:
                pass

2. Add deep_crawl_strategy handling in arun_many (around line 1058):

# Add after config is set, before stream handling
if primary_config and primary_config.deep_crawl_strategy:
    if stream:
        # For streaming: merge multiple async generators
        generators = []
        for url in urls:
            generators.append(
                primary_config.deep_crawl_strategy.arun(
                    start_url=url, crawler=self, config=primary_config
                )
            )
        return merge_async_generators(generators)
    else:
        # For batch mode: collect results per URL
        results = []
        for url in urls:
            url_results = []
            async for result in primary_config.deep_crawl_strategy.arun(
                start_url=url, crawler=self, config=primary_config
            ):
                url_results.append(result)
            results.append(url_results)
        return results

[vikas-gits-good]

I had made a pull request that was closed that solves this bug. I made a new pull request that is open. Please assign this to me.