Release v0.1.6

📦 Published on PyPI: https://pypi.org/project/crawl4ai-mcp-llm/0.1.6/

🆕 max_pages option for the crawl tool (#40)

BFSDeepCrawlStrategy now accepts max_pages, exposed as a crawl tool argument. max_pages=1 fetches exactly one page without following any link — the safe way to grab a single documentation page.

Previously, crawling a single doc page required stripping <a> elements from the DOM via js_code to prevent link fan-out. That silently deleted every linked term from the output (type names, cross-references) and corrupted API-doc crawls.

Full Changelog: v0.1.5...v0.1.6

v0.1.5 — MCP 2.0.0 stateless protocol

📦 Published on PyPI: https://pypi.org/project/crawl4ai-mcp-llm/0.1.5/

ℹ️ This release and its tag were created retroactively (Aug 31, 2026). The package was published to PyPI on Aug 1, 2026 from this exact commit — the only difference is the version string, bumped to 0.1.5 as a local-only change at the time.

🔄 Migration to MCP SDK 2.0.0 — stateless protocol (#38)

Updated the MCP dependency to 2.0.0 and migrated the server to the stateless protocol.

🐛 Fixes

• Redirected print statements to stderr — stdout must stay reserved for the MCP JSON-RPC protocol, stray prints were breaking client communication.

📄 License

• Added the MIT License to the project.

Full Changelog: v0.1.4...v0.1.5

Release v0.1.4

Full Changelog: v0.1.3...v0.1.4

v0.1.3 — Code review fixes

📦 Published on PyPI: https://pypi.org/project/crawl4ai-mcp-llm/0.1.3/

🔍 Code review fixes

This release addresses issues found during a full code review of the project.

Bugs fixed

• max_depth validation (crawler.py): max_depth=0 was ambiguous with BFSDeepCrawlStrategy and could trigger unexpected crawling. Now enforced to >= 1 (use 1 for a single page).
• Error detection regex (crawler.py): ERROR_INDICATORS_REGEX was too broad and could flag legitimate pages (e.g. an article titled "Understanding HTTP 404 errors") as errors, silently skipping them. Now anchored to match only actual error-page titles.

Robustness

• Runtime type coercion (server.py): MCP clients (LLMs) may send strings instead of the declared JSON-schema types. Parameters are now cast defensively (int, float, bool) to avoid cryptic failures deep in crawl4ai.

Quality

• DRY (crawler.py): Extracted _empty_stats() to remove the 3× duplicated stats dict across error paths.
• Renamed sanitize_text → sanitize_for_display with a docstring warning that it destroys typographic Unicode (not for crawled content).
• Removed 9 temporary dev artifacts from the repo root (patch_*.py, test_crawl_local*.py, correction_plan.md, benchmark_links.py, mcp_server.log).

Verified

• ✅ 43 tests pass (0 failures)
• ✅ Build OK — complete wheel
• ✅ Install from PyPI tested end-to-end

Full Changelog: v0.1.2...v0.1.3

v0.1.2

⚡ Performance warning in the tool description

The crawl tool now exposes a performance warning in its MCP description, visible to any MCP client (ZCode, Claude Desktop, Cline) via tools/list. This lets the AI warn users before launching a potentially slow crawl.

What's new

• Tool description updated (src/crawl4ai_mcp_llm/server.py):
  • Warns that crawls can take 30 seconds to several minutes
  • Recommends setting the MCP client timeout generously (e.g. 600000 ms / 10 min)
  • Lists tips to speed up crawls:
    • css_selector to extract only the relevant content (main, article)
    • wait_for_selector for single-page applications
    • Lower max_depth when recursive crawling isn't needed

Why

Heavy/SPA sites (React, Next.js, Mintlify), high max_depth, and the first crawl of a session (Playwright browser startup) can be slow. Without this warning, the AI may launch a crawl without informing the user it could take minutes.

Install

uvx crawl4ai-mcp-llm

Full Changelog: v0.1.1...v0.1.2

v0.1.1

📦 Published on PyPI

This release ships the package on PyPI under the new name crawl4ai-mcp-llm.

What's new since v0.1.0

• Installation from PyPI is now the recommended method — no need for --from git+...:

  uvx crawl4ai-mcp-llm

• Dedicated PyPI README (README_PYPI.md) with an absolute image URL and no broken language badges, so the PyPI project page renders correctly.
• All 9 localized READMEs (ar, bn, es, fr, hi, id, pt, ru, zh) have been fully retranslated from the corrected source — they no longer reference the old package name or the git-only install path.
• The GitHub README.md keeps the language badges and now documents both install methods (PyPI recommended + git alternative).

Install

From PyPI (recommended):

{
  "mcpServers": {
    "crawl": {
      "command": "uvx",
      "args": ["--python", "3.13", "crawl4ai-mcp-llm"],
      "timeout": 600
    }
  }
}

From GitHub:

{
  "mcpServers": {
    "crawl": {
      "command": "uvx",
      "args": ["--python", "3.13", "--from", "git+https://github.com/laurentvv/crawl4ai-mcp-llm", "crawl4ai-mcp-llm"],
      "timeout": 600
    }
  }
}

Verified

• ✅ Install from PyPI via uvx crawl4ai-mcp-llm tested end-to-end
• ✅ Server answers the MCP handshake (serverInfo: mcp-web-crawler v1.28.1)
• ✅ No broken image or language links in any README

Full Changelog: v0.1.0...v0.1.1

v0.1.0 — Renamed to crawl4ai-mcp-llm

🔄 Project renamed: crawl4ai-mcp → crawl4ai-mcp-llm

Background

The PyPI package name crawl4ai-mcp does not belong to this project. It was registered by a third party, and the published package is broken — the crawl4ai_mcp module is missing from the distribution, causing a ModuleNotFoundError at startup.

To avoid the conflict and publish under a name we control, the project has been renamed to crawl4ai-mcp-llm across the entire chain: GitHub, PyPI, Python module, and executable.

📦 Published on PyPI: https://pypi.org/project/crawl4ai-mcp-llm/

Changes

Item	Before	After
GitHub repo	laurentvv/crawl4ai-mcp	laurentvv/crawl4ai-mcp-llm
PyPI package	(squatted, unusable)	crawl4ai-mcp-llm
Python module	crawl4ai_mcp	crawl4ai_mcp_llm
Executable	crawl4ai-mcp	crawl4ai-mcp-llm
Results directory	~/.crawl4ai_mcp/	~/.crawl4ai_mcp_llm/

ℹ️ The old repository github.com/laurentvv/crawl4ai-mcp automatically redirects to the new one. Existing clones and forks keep working.

Installation

From PyPI (recommended):

{
  "mcpServers": {
    "crawl": {
      "command": "uvx",
      "args": [
        "--python", "3.13",
        "crawl4ai-mcp-llm"
      ],
      "timeout": 600
    }
  }
}

From GitHub (latest unreleased):

{
  "mcpServers": {
    "crawl": {
      "command": "uvx",
      "args": [
        "--python", "3.13",
        "--from", "git+https://github.com/laurentvv/crawl4ai-mcp-llm",
        "crawl4ai-mcp-llm"
      ],
      "timeout": 600
    }
  }
}

⚠️ For existing users

The results directory has changed: ~/.crawl4ai_mcp/results → ~/.crawl4ai_mcp_llm/results. Your old Markdown files remain on disk in the old path but are no longer visible to the crawler. Move them manually if needed:

mv ~/.crawl4ai_mcp/results ~/.crawl4ai_mcp_llm/results

Verification

• ✅ uv build produces a complete wheel (module included)
• ✅ Server starts and answers the MCP handshake (serverInfo: mcp-web-crawler v1.28.1)
• ✅ Install from PyPI via uvx crawl4ai-mcp-llm verified end-to-end
• ✅ All internal imports and tests updated

---

Full Changelog: crawl4ai-mcp → crawl4ai-mcp-llm (full rename, no functional changes)

Adding filter to crawl

Filtered Crawl Process

The crawl process has been refined to ensure that only relevant content is retained. Here's a detailed breakdown of the steps involved:

Initial Crawl:

The crawler begins by fetching web pages based on the initial set of URLs.
Content Extraction:

Extract the main content from each page, focusing on textual information.
Maintain the structure of the content, including headings, paragraphs, and lists.
Link Removal:

Identify and remove all hyperlinks from the extracted content.
Ensure that the removal of links does not disrupt the overall structure or readability of the text.
Code Structure Preservation:

Preserve the Python indentation and code structure within the markdown.
Ensure that code blocks are properly formatted and indented for clarity.
Final Markdown Generation:

Compile the filtered content into a markdown format.
The final markdown will be concise, with a focus on readability and structure, free from extraneous links.

First Functional Version

I am excited to deploy the first version of my MCP, designed to retrieve Markdown documentation, primarily for Visual Studio Code. As a beginner, I welcome any constructive feedback and suggestions to improve this tool. Feel free to test it out and share your thoughts!