v0.33.2

• The PyPI project page links to the project website, https://www.searxncrawl.io/, like the MCP Registry entry and the MCPB bundle.
• No runtime changes compared with 0.33.1.

Related pull request: #77.

v0.33.1

• Fix Chromium zombie accumulation in the native and optional Arcade Docker images with a Tini entrypoint.
• Verify repeated MCP browser calls in both images in CI, including parallel crawl, site crawl and search with page content.
• Add the optional Arcade adapter and refresh native installation documentation.
• Upgrade: rebuild and recreate repository containers. Custom containers need --init, Compose init: true or Podman Quadlet RunInit=true; upgrading the Python package alone does not add init.
• Keep the image entrypoint when overriding commands; --entrypoint bypasses Tini.

Related issue: #72.

v0.33.0

• Add validated SearXNG extra headers for CLI and MCP, including absolute/~/ secret-file references. Invalid configuration disables only SearXNG; credential values stay out of diagnostics.
• Add search with page content in one call (include_content / --include-content). Nest crawl Markdown headings under page/result headings; JSON Markdown is unchanged.
• Refresh README and usage/module/registry documentation; expose basic auth, extra headers and result fields in MCPB configuration.
• Refresh locked dependencies and pinned publishing actions.

Validation:

• 42 live CLI/MCP STDIO/HTTP cases with real SearXNG and You.com, protected-header gateway and public-page browser crawls.
• 259 required-browser tests, Ruff, wheel/sdist metadata and registry/MCPB schema checks pass.
• Original contribution from KuehhornIT's #48 is retained in the attribution; replacement #67 is merged.

Known limitation:

• The Python 3.10 lock path still selects dnspython 2.8.0; Dependabot cannot apply 2.9.0 because it requires Python >=3.11. Python support is unchanged.

Distribution status:

• Published on PyPI and the official MCP Registry as 0.33.0. Native and Smithery MCPB bundles are attached below.
• Smithery publishing currently returns HTTP 403, "Updating servers has been removed". Smithery announced API, account and CLI shutdown for 31 October 2026. Use direct PyPI installation or the native MCPB bundle; the legacy Smithery variant remains attached for compatibility. The curated Smithery listing has not been updated.

v0.32.0

Upgrade note: existing SearXNG-only setups must set SEARXNG_URL explicitly; there is no http://localhost:8888 default any more. SEARXNG_URL must not embed credentials (https://user:pw@host is rejected); use SEARXNG_USERNAME/SEARXNG_PASSWORD instead.

Added

• Multi-provider search: CLI search and MCP search query every configured provider (SearXNG, You.com) in parallel and return one merged list (conservative URL dedup, round-robin, results found by several providers first). A partial failure still returns the other providers' results.
• You.com provider, enabled by YDC_API_KEY (keyed API only; queries go to You.com, US). Requests use POST /v1/search with a JSON body. safesearch maps to off/moderate/strict; language maps to You.com's supported codes (en-us → EN; unsupported codes such as zh search without a language filter instead of failing with HTTP 422); pages beyond 10 return no You.com results; content falls back to the first snippet when the description is empty. HTTP 403 is reported as a missing key scope, separate from 401.
• Provider selection per call: MCP providers parameter, CLI --providers searxng,youcom.
• providers field on every result; the CLI markdown output shows it as a **Sources:** line.
• searxncrawl setup registers the MCP server with the installed coding agents (Claude Code, Codex, Gemini CLI, Cursor, VS Code, Claude Desktop) in one step. It shows each change, asks before applying it (--yes skips the prompt), backs up every file it touches, updates instead of duplicating on re-runs, and setup --remove unregisters again.
• searxncrawl setup can also store a You.com API key (--ydc-api-key, or a hidden prompt suggesting $YDC_API_KEY). The key is masked as <hidden> in everything setup prints, and setup warns when no search provider is configured.
• server.json for the official MCP Registry (io.github.DasDigitaleMomentum/searxncrawl). The release workflow now fails if its versions differ from pyproject.toml. See docs/registry.md.

Changed

• Breaking: no SEARXNG_URL default any more (code, docker-compose.yml, .env.example); SearXNG is active only when SEARXNG_URL is set. Setups that relied on http://localhost:8888 must set it explicitly.
• Breaking: SEARXNG_URL must be an http(s) URL without embedded credentials (https://user:pw@host and scheme-less values are rejected with a warning); use SEARXNG_USERNAME/SEARXNG_PASSWORD instead, both or neither.
• Breaking: unset SEARCH_RESULT_FIELDS now returns the normalized fields title,url,content,publishedDate,providers; the raw SearXNG payload passthrough ends, so fields such as engine, score, suggestions, answers are gone. Add providers to an existing SEARCH_RESULT_FIELDS to show attribution.
• Breaking: the CLI "Related searches" line (from SearXNG suggestions) is removed.
• Search errors are short per-provider reasons without secrets, URLs or response bodies (a SearXNG non-401 status no longer includes the response body).
• The Docker image is based on python:3.14-slim instead of python:3.11-slim.
• README rewritten around a demo, one-line agent setup and a comparison; installation, MCP server, CLI and Python API reference moved to docs/usage/.
• HTTP 429 from a search provider is retried once when Retry-After is at most 5 s and attempts remain (MCP max_retries; the CLI makes a single attempt and never retries). Otherwise the reason ends with retry after Ns or try again later.

Fixed

• Pages could be captured mid-reload: the default config ran window.location.reload() in js_code, which Crawl4AI executes after wait_for without waiting for the new document. On docs.python.org this returned only <head> and failed as "Blocked by anti-bot protection". The reload now runs as a Playwright after_goto hook (page.reload(wait_until="load")), so wait_for and capture see the reloaded page. Custom configs and js_code overrides keep their previous behaviour (no reload).
• Content rendered into shadow DOM (e.g. MDN code examples) was missing from the markdown; flatten_shadow_dom is now enabled. Raises the minimum Crawl4AI version to 0.8.5, the first release with this option.
• Code samples lost < and </ in the markdown: PruningContentFilter dropped Prism/highlight.js/CodeMirror token spans holding only those characters, so <details> became details. <pre>, <code> and CodeMirror's .cm-content are now exempt from pruning. Raises the minimum Crawl4AI version to 0.9.1, the first release with preserve_tags/preserve_classes.
• Every crawl failed with CrawlerRunConfig.__init__() got an unexpected keyword argument 'flatten_shadow_dom' when the code ran against Crawl4AI < 0.8.5 (e.g. a stale virtualenv). flatten_shadow_dom is now passed only if the installed Crawl4AI supports it; otherwise a warning is logged and shadow DOM content is skipped.
• Shadow DOM flattening corrupted pages whose text contains <, > or &: Crawl4AI's flatten_shadow_dom.js writes text nodes back unescaped, so text such as <details> was re-parsed as markup. Closing tags vanished from HTML code samples, and the broken tree defeated target_elements, pulling site navigation into the markdown (MDN: 38 kB with nav instead of 22 kB of article). The script is now patched at import to escape text nodes.

Security

• nltk is no longer installed from uv.lock or in the Docker image. It has an unpatched advisory (GHSA-8mgp-746c-j5xp, all versions up to 3.10.3) and comes in only through crawl4ai, which imports it in chunking and legacy helpers that searxNcrawl does not use. pip/uvx installs from PyPI still pull it in through crawl4ai's metadata; the affected functions are not called there either.

Install: uvx searxncrawl (MCP server) · uvx searxncrawl setup (register with your coding agents) · pip install searxncrawl

v0.31.0

Added

• Zero-setup runs with uvx: uvx --from git+https://github.com/DasDigitaleMomentum/searxNcrawl searxncrawl starts the MCP server without a clone or virtualenv. (refs #14)
• Playwright's Chromium is installed automatically once per process before the first browser launch if it is missing (python -m playwright install chromium, idempotent; output goes to stderr so MCP stdio stays clean). Disable with PLAYWRIGHT_AUTO_INSTALL=false. (refs #14)
• searxncrawl console script as an alias for crawl-mcp, so the package name doubles as the MCP server command.

Fixed

• Single-page crawls no longer time out on pages without a <main> element (e.g. example.com). The wait_for condition now accepts any MAIN_SELECTORS area (article, [role='main'], .markdown-body, …) with more than 50 characters and falls back to the page body after a 3 s grace period once the document has loaded, instead of waiting for the 30 s page timeout.

Install: uvx searxncrawl (MCP server) · pip install searxncrawl

v0.30.0 — HTTP Host & Origin Configuration

What's new

• Added --allowed-hosts for trusted HTTP Host-header configuration in remote, Docker, and reverse-proxy deployments.
• Documented FastMCP's existing FASTMCP_HTTP_ALLOWED_HOSTS JSON-list environment setting.
• Wired --cors-origins into both FastMCP's request-time Origin guard and Starlette CORS middleware.
• Preserved FastMCP's secure Host/Origin defaults and kept wildcard access as an explicit opt-in.
• Raised the FastMCP minimum version to 3.4.3.
• Added real ASGI guard coverage plus CLI/runtime and STDIO-isolation tests.

Verification

• 99 tests passed
• Package metadata verified as 0.30.0

Issue closed

• #25 — HTTP transport unreachable from non-localhost clients

v0.25.0 — SEARCH_RESULT_FIELDS & LOG_LEVEL

What's new

• SEARCH_RESULT_FIELDS env var: control which fields appear per search result in MCP + CLI. Default recommended: title,url,content,publishedDate. Unset = all SearXNG fields (backward-compatible).
• LOG_LEVEL env var: control MCP server log verbosity (DEBUG, INFO, WARNING, ERROR, CRITICAL). Controls both root logger and uvicorn/FastMCP.
• Version logging at MCP server startup
• MCP crawl() timeout reduced from 30s to 15s for faster feedback
• README: corrected Docker Compose SearXNG docs, added env var examples

Issues closed

• #23 — "Short" tool option (SEARCH_RESULT_FIELDS)
• #24 — Expose some logging knobs (LOG_LEVEL)

v0.24.2

What's Changed

Changed

• README restructured — slimmed from 731 to 291 lines with Quick Start, installation feature matrix, and compact sections.
• Reference docs extracted to docs/usage/:
  • mcp-tools.md — full tool parameter reference
  • output-formats.md — Markdown/JSON output + CrawledDocument
  • session-capture.md — crawl-capture guide
• Docker Compose promoted to first-class installation path with SearXNG included.

v0.24.1

What's Changed

Fixed

• Corrected BFS→DFS documentation references in README, MCP tool docstring, CLI help text, and module docstring to match the actual DFSDeepCrawlStrategy implementation (closes #18).

v0.24.0

What's Changed

Fixed

• MCP stdio protocol noise — Crawl4AI verbose defaults disabled in build_markdown_run_config() and build_discovery_run_config() to prevent protocol corruption from noisy init output.
• UTF-8 stdio encoding — Forced UTF-8 with replacement error handling on stdio transport for Windows and non-UTF-8 locale compatibility.