[Bug]: PDFCrawlerStrategy fails with "Blocked by anti-bot protection" - the documented PDF example is broken

[SohamKukreti]

crawl4ai version

0.9.2 (reproduced on latest develop, ea26abb)

Expected Behavior

Running the exact example from the docs (Advanced → PDF Parsing, docs/md_v2/advanced/pdf-parsing.md) should return result.success = True with the extracted PDF content, as the page describes.

Current Behavior

The documented PDFCrawlerStrategy + PDFContentScrapingStrategy pairing always returns success: False with:

Blocked by anti-bot protection: Near-empty content (33 bytes) with HTTP 200

— on every PDF, every time. The extraction itself actually succeeds: the same CrawlResult carries the complete PDF markdown (25,297 chars for the docs' example URL). The result is self-contradictory — full content plus a failure flag — so any caller that checks result.success (as the docs example does) discards a perfectly good scrape.

Root cause

1. PDFCrawlerStrategy.crawl() returns placeholder HTML — literally "Scraper will handle the real work", 33 bytes (crawl4ai/processors/pdf/__init__.py:17). The real extraction happens later in the scraping strategy, which populates cleaned_html/markdown from the PDF.
2. is_blocked() only ever sees this pre-scrape placeholder, never the scraped output. Verified by instrumenting is_blocked(): it fires twice per crawl, both times with the 33-byte stub —
   • once inside the attempt loop (crawl4ai/async_webcrawler.py:512), which classifies the attempt as blocked — so with retries/proxy_rotation_strategy configured, every proxy and retry is burned on a crawl that is actually succeeding, and a configured fallback_fetch_function gets invoked pointlessly;
   • once in the final post-processing veto (crawl4ai/async_webcrawler.py:629), which is what flips success = False.
3. 33 stripped bytes with HTTP 200 trips the near-empty-content heuristic (crawl4ai/antibot_detector.py:274).

Notably, the code already exempts three cases where this heuristic misfires — successful fallback fetches, raw: URLs, and binary downloads ("is_blocked() would misread '0 bytes html' as a block", async_webcrawler.py:619-628). A PDFCrawlerStrategy crawl is a fourth case of the same shape: html is a stub by design, and the content lives elsewhere. It matches none of the existing exemptions (downloaded_files is not set; the PDF goes through a temp file).

Is this reproducible?

Yes — deterministic, 100% of PDFs. Also reproduces with local file:// PDFs.

Inputs Causing the Bug

Any arun() call through AsyncWebCrawler(crawler_strategy=PDFCrawlerStrategy()).

Steps to Reproduce

1. pip install crawl4ai[pdf]
2. Run the example from the PDF parsing docs page, unmodified.
3. Observe Failed to process PDF: Blocked by anti-bot protection: Near-empty content (33 bytes) with HTTP 200 — while result.markdown.raw_markdown contains the full extracted paper.

Code snippets

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy

async def main():
    # Exact example from docs/md_v2/advanced/pdf-parsing.md
    pdf_crawler_strategy = PDFCrawlerStrategy()
    pdf_scraping_strategy = PDFContentScrapingStrategy()
    run_config = CrawlerRunConfig(scraping_strategy=pdf_scraping_strategy)

    async with AsyncWebCrawler(crawler_strategy=pdf_crawler_strategy) as crawler:
        pdf_url = "https://arxiv.org/pdf/2310.06825.pdf"
        result = await crawler.arun(url=pdf_url, config=run_config)
        if result.success:
            print(f"Successfully processed PDF: {result.url}")
        else:
            print(f"Failed to process PDF: {result.error_message}")
            md = result.markdown.raw_markdown if hasattr(result.markdown, "raw_markdown") else result.markdown
            print(f"...yet extracted markdown length is: {len(md or '')}")

Output:

[SCRAPE].. ◆ https://arxiv.org/pdf/2310.06825.pdf  | ✓ | ⏱: 1.07s
[ERROR]... × https://arxiv.org/pdf/2310.06825.pdf  | Error: Blocked by anti-bot protection: Near-empty content (33 bytes) with HTTP 200
Failed to process PDF: Blocked by anti-bot protection: Near-empty content (33 bytes) with HTTP 200
...yet extracted markdown length is: 25297

OS

Linux

Python version

3.10

Related

• Config option to disable is_blocked() post-crawl content veto #2058 — general request for a config option to disable the is_blocked() post-crawl veto (this report is a deterministic instance of that mechanism misfiring on a documented first-party workflow)
• Workaround for anyone hitting this: use AsyncWebCrawler(crawler_strategy=AsyncHTTPCrawlerStrategy()) with PDFContentScrapingStrategy — verified working (full extraction, success: True), since the HTTP strategy returns the real PDF bytes as the response body and the heuristic never fires.

[nightcityblade]

Hi, I'd like to work on this. I'll submit a PR shortly.

[SohamKukreti]

Hi @nightcityblade, thanks for picking up #2135 so quickly and for catching both call sites (the attempt-loop classification and the final veto, with the _is_pdf_response = False init before the loop). The header extraction is also nicely defensive (case-insensitive, str(value)).

I explored this same content-type-based design while preparing my own fix and moved away from it after review, I've opened #2138 with the alternative, and I want to lay out the trade-offs honestly so we can pick with full information.

Why I didn't go with content-type sniffing

1. It hardcodes one media type into the core crawler. The reason this response deserves an exemption isn't "it's a PDF" — it's "its html is a placeholder, not fetched page content" (the real content arrives via the scraping strategy). Any future strategy with the same shape (DOCX/EPUB processors, custom strategies that defer extraction) needs another special case edited into the same two guards in async_webcrawler.py. My PR adds AsyncCrawlResponse.placeholder_html (default False) so any strategy can self-declare — mirroring the existing raw:-URL and downloaded_files exemptions on those exact guard lines, with zero core changes needed for the next strategy.

2. Test coverage of the real workflow. Monkeypatching crawler.aprocess_html neatly avoids the pypdf dependency, and the assertions do bite on pre-fix code — but it means the documented pairing (PDFCrawlerStrategy + PDFContentScrapingStrategy) is never exercised and no extracted content is ever asserted. My PR tests the full pipeline against a generated local PDF fixture (no network, pytest.importorskip("pypdf") so envs without the extra skip instead of erroring), plus negative controls verifying that near-empty HTML pages are still vetoed and that retries/fallback_fetch_function stay quiet with max_retries set.

Your PR is also correct and a good approach as well! Either of the two will work.
Thanks!