[Bug]: Docker /crawl skip the content

[RotEveryDay]

crawl4ai version

docker images unclecode/crawl4ai:0.6.0-rN

Expected Behavior

I hope to be able to scrape the title: "Product Safety Information Notice on a LED UV Nail Lamp sold on AliExpress", I can scrape it directly by calling arun(), but when using docker this content does not exist.

url:https://www.ccpc.ie/consumers/product_recalls/product-safety-information-notice-on-a-led-uv-nail-lamp-sold-on-aliexpress/

Current Behavior

no information about"Product Safety Information Notice on a LED UV Nail Lamp sold on AliExpress"

Is this reproducible?

Yes

Inputs Causing the Bug

curl -X POST http://localhost:11235/crawl \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://www.ccpc.ie/consumers/product_recalls/product-safety-information-notice-on-a-led-uv-nail-lamp-sold-on-aliexpress/"],"crawler_config":{"type":"CrawlerRunConfig","params":{"scraping_strategy":{"type":"WebScrapingStrategy","params":{}},"exclude_social_media_domains":["facebook.com","twitter.com","x.com","linkedin.com","instagram.com","pinterest.com","tiktok.com","snapchat.com","reddit.com"]}}}'112

Steps to Reproduce

Code snippets

OS

Linux

Python version

as docker

Browser

Chrome

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[PhantasWeng]

Hi, @RotEveryDay

I encountered the same issue on other websites as well.

I just tested it using the link you provided.
Did some testing using other configs like wait_for, page_timeout, and delay_before_return_html didn’t work.

But! Setting scan_full_page=True successfully allowed me to retrieve the title Product Safety Information Notice on a LED UV Nail Lamp sold on AliExpress

# CrawlerRunConfig
# Document: https://docs.crawl4ai.com/api/parameters/

CrawlerRunConfig(
    stream=True,
    cache_mode=CacheMode.BYPASS,
    scan_full_page=True
)

Unfortunately, the URL I want still cannot retrieve information through these configurations.
This is the URL I want to know: https://www.nasdaq.com/market-activity/ipos
I am trying to get the upcoming data list

[RotEveryDay]

@PhantasWeng Thank you very much! You indeed solved my problem, but I'm sorry I couldn't help you with yours.

[RotEveryDay]

@PhantasWeng BTW,I encountered a new issue while using Docker.

url="https://worldcrunch.com/world-affairs/mexico-united-states-nafta-trade/"
I found that when I use Docker to crawl, it would be blocked.

then I choose to crawl by:

import asyncio
import re
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(
            url="https://worldcrunch.com/world-affairs/mexico-united-states-nafta-trade/",
            cache_mode=CacheMode.BYPASS
        )
        print(result.markdown)
        data = result.markdown

        with open("tmp.txt", "w", encoding="utf-8") as file:
            file.write(data)

if __name__ == "__main__":
    asyncio.run(main())

It can get what I want!