Am I using CrawlerRunConfig parameters correctly?

[pliny-TheElder]

I'm trying to scrape a site to build an LLM knowledge base for a specific use-case I'll be working with. When I run the below code, none of the parameters seem to apply and I get the same return markdown whether I run with css_selector, excluded_tags, text_only, etc. with or without wait_for. I've also tried many different formats for the css_selector and wait_for values = .segment-content, #segment-content, css: .segment-content, etc. and none work. Lastly, I've tried various outputs like cleaned_html, markdown, markdown_v2, etc.

#segment-content

Can someone take a look at this site and see if I'm doing something wrong? I'm really new to web scraping but understand HTML. This seems like a very straightforward use-case and I've compared to many examples applying parameters and they're not working.

I also can't seem to figure out the debug browser (headless=False) as it doesn't open as I'd expect when I run the script.

Site I'm trying: https://hl7.org/fhir/R5/

import asyncio
from crawl4ai import AsyncWebCrawler
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    # Browser configuration
    browser_config = BrowserConfig( 
        browser_type="chromium",
        text_mode=True, 
        headless=False,
        verbose=True, )

    # Run configuration
    run_config = CrawlerRunConfig(
        wait_for="css:#segment-content",  # Wait for the main content to load
        css_selector="#segment-content",  # Only include this section
        markdown_generator=DefaultMarkdownGenerator(),  # Generate markdown
        only_text=True,  # Extract only text
        exclude_external_links=True,  # Remove external links
        exclude_social_media_links=True,  # Remove social media links
        remove_overlay_elements=True,  # Remove popups/modals
        page_timeout=30000  # Set page timeout to 30 seconds (in milliseconds)
    )

    # Initialize the crawler
    async with AsyncWebCrawler(browser_config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://hl7.org/fhir/R5/",
            config=run_config
        )
        if result.success:
            # Print the cleaned markdown content
            
            print(result.markdown)
            print(len(result.markdown))
        else:
            print("Crawl failed:", result.error_message)
  
# Run the async main function
if __name__ == "__main__":
    asyncio.run(main())

[rserdah01]

I have/had the same problem, but I ran playwright install and it seemed to fix it, but I am still messing with it.

If it says playwright is not recognized, then you may have to change directory where playwright is and run the command there. For me on Windows, it was C:\Users\<User>\AppData\Roaming\Python\Python312\Scripts, but I installed it for user only so this may not be the same for you.

If on Windows and playwright is not a global command, after changing to the directory, you may have to run the command like .\playwright.exe install with slashes maybe forward slash, backslash, or just do playwright.exe install if it works.