[Bug]: memory leak issue when running the crawler in Docker

[fred-mhyt]

crawl4ai version

0.7.4

Expected Behavior

Concurrent url crawling

Current Behavior

I encountered a memory leak issue when running the crawler in Docker. Our production server configuration is 1 CPU and 8GB of memory, and even when crawling only 5 URLs concurrently, the memory leak still occurs. However, it works normally on Windows, and the memory_threshold_percent parameter in MemoryAdaptiveDispatcher has no effect at all.

default_memory_dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=85,
    check_interval=1,
    max_session_permit=5,
    rate_limiter=RateLimiter(
        base_delay=(3, 15), 
        max_delay=20.0,
        max_retries=1 
    )
)

 async with AsyncWebCrawler(config=self.general_browser_config) as crawler:
                results = await crawler.arun_many(
                    urls=urls,
                    config=config,
                    dispatcher=default_memory_dispatcher 
                )

This is the memory usage statistics in production. The more complex the content of a web page is, the more memory it uses

This is the local windows memory usage statistics

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

OS

Linux（docker）

Python version

3.12.0

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[fred-mhyt]

To address this issue, I temporarily wrote a custom memory monitoring method. When the memory usage exceeds the 80% threshold, the crawling tasks are directly canceled, which however causes the failure of the URLs being crawled at that time.

[SohamKukreti]

Hey, we have already worked on something addressing this issue. It will be released soon!
PR: #1595

[fred-mhyt]

Hey, we have already worked on something addressing this issue. It will be released soon! PR: #1595

Hey, we have already worked on something addressing this issue. It will be released soon! PR: #1595

Hello, I've seen your modifications and really appreciate your contributions. However, I'd like to ask: do these changes only take effect for the Docker image version of Crawl4AI? I'm using Crawl4AI in a project within a Docker environment, and I installed it via the pip install Crawl4AI method.

[SohamKukreti]

@fred-mhyt Just to confirm you are facing memory leak issues in the library installed via pip within a docker container?
Can you share a bit more about how you have set it up?

[fred-mhyt]

@fred-mhyt Just to confirm you are facing memory leak issues in the library installed via pip within a docker container? Can you share a bit more about how you have set it up?

general_browser_config = BrowserConfig(
    user_agent_mode="random",
    enable_stealth=True,
    viewport_height=1080,
    viewport_width=1920,
    headless=True,
    verbose=False,
)

general_crawler_config = CrawlerRunConfig(
    markdown_generator=DefaultMarkdownGenerator(
        content_source="fit_html",
    ),
    cache_mode=CacheMode.DISABLED,
    screenshot=True,
    wait_for_images=False,
    exclude_external_images=True,
    process_iframes=False,
    scan_full_page=False,
    session_id=None,
    pdf=False,
    verbose=False,
    simulate_user=False,
    delay_before_return_html=1,
    page_timeout=6000,  # ms
    only_text=True,
    semaphore_count=2,
    excluded_tags=["script", "style"],
    remove_overlay_elements=True,
)

use arun_many，the configuration is 2 cpus and 8GB of memory

default_memory_dispatcher = MemoryAdaptiveDispatcher(
    memory_threshold_percent=85,
    check_interval=1,
    max_session_permit=3,
    rate_limiter=RateLimiter(
        base_delay=(1, 3), 
        max_delay=10.0,  
        max_retries=1 
    )


    async def crawl_batch_with_arun_many(self, urls: List[str], deep_crawl: bool = True) -> List[
        Tuple[str, Optional[CrawlResult]]]:
        
        if not urls:
            return []

        config = self.general_crawler_config if deep_crawl else self.google_crawler_config

        try:
            async with AsyncWebCrawler(config=self.general_browser_config) as crawler:
                results = await crawler.arun_many(
                    urls=urls,
                    config=config,
                    dispatcher=self.dispatcher_by_semaphore
                )

                paired_results = []
                success_count = 0

                for i, url in enumerate(urls):
                    result = results[i] if i < len(results) else None

                    if result and result.success:
                        success_count += 1
                        
                    elif result:
                        result = None
                    paired_results.append((url, result))

                return paired_results

        except Exception as e:
            return [(url, None) for url in urls]

After using arun_many memory overflow, I continued to use the following batch run method

    async def crawl_batch_with_arun(
            self,
            urls: List[str],
            deep_crawl: bool = True,
            max_concurrency: Optional[int] = None
    ) -> List[Tuple[str, Optional[CrawlResult]]]:

        if not urls:
            return []

        config = self.general_crawler_config if deep_crawl else self.google_crawler_config

        if max_concurrency is None:
            max_concurrency = crawler_configs.max_session_permit
        max_concurrency = max(2, int(max_concurrency))

        paired_results: List[Tuple[str, Optional[CrawlResult]]] = []
        success_count = 0

        try:
            for i in range(0, len(urls), max_concurrency):
                batch_urls = urls[i:i + max_concurrency]
                crawler_logger.info(f"psutil.virtual_memory: {psutil.virtual_memory()},{psutil.virtual_memory().percent}")
                async with AsyncWebCrawler(config=self.general_browser_config) as crawler:
                    async def run_one(u: str):
                        return await crawler.arun(url=u, config=config)

                    tasks = [asyncio.create_task(run_one(u)) for u in batch_urls]
                    
                    results_raw = await asyncio.gather(*tasks, return_exceptions=True)
                   
                   
                    gc.collect()

                for j, url in enumerate(batch_urls):
                    result = results_raw[j] if j < len(results_raw) else None

                    if result is not None and not isinstance(result, Exception):
                        size_bytes = int(asizeof(result))
                        size_mb = size_bytes / (1024 * 1024)

                    if isinstance(result, Exception):
                        continue

                    if result and getattr(result, 'success', False):
                        success_count += 1
                        paired_results.append((url, result))
                    else:
                        crawler_logger.warning(f" {url}")
                        # paired_results.append((url, None))

                await asyncio.sleep(1)

            return paired_results

        except Exception as e:
            return [(url, None) for url in urls]

[SohamKukreti]

@fred-mhyt Thanks for sharing. I will look at this and get back to you soon.