MCP Web Search

MCP server for web search and URL/resource loading. It works without API keys by default and stays local-first: search uses free providers, fetch_url extracts useful content from URLs, and binary/media downloads only happen when explicitly requested.

Features

• search_web - multi-provider web search with automatic fallback across DuckDuckGo, Bing, and SearXNG.
• fetch_url - universal URL/resource loader for HTML, PDF, text, Markdown, JSON, XML, CSV, media metadata, and supported site-specific URLs.
• Clean normalized output with one content field plus metadata, pagination, links, media, attachments, and warnings.
• Reddit thread extraction through Reddit JSON endpoints instead of brittle Reddit HTML scraping.
• Long-resource pagination with max_length, start_index, and next_start_index.
• Optional HTML link/media summaries.
• Optional local download artifacts with download: true.
• SSRF protection for localhost, private IPs, link-local ranges, IPv6 private ranges, and unsafe redirects.
• No paid API required.

Requirements

• Node.js 18+
• Chrome/Chromium only if you use the Bing provider

MCP Configuration

Claude Code

{
  "mcpServers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "@zhafron/mcp-web-search"]
    }
  }
}

OpenCode

{
  "mcp": {
    "web-search": {
      "type": "local",
      "command": ["npx", "@zhafron/mcp-web-search"]
    }
  }
}

Custom Configuration

{
  "mcpServers": {
    "web-search": {
      "command": "npx",
      "args": ["-y", "@zhafron/mcp-web-search"],
      "env": {
        "DEFAULT_SEARCH_PROVIDER": "duckduckgo",
        "SEARXNG_URL": "http://localhost:8099"
      }
    }
  }
}

Tools

search_web

Search the web through one provider or through the fallback chain.

Input:

{
  "q": "openai codex reddit review",
  "limit": 10,
  "lang": "en",
  "provider": "duckduckgo"
}

Options:

Option	Description
q	Search query
limit	Number of results, 1-50
lang	Search language, default en
provider	Optional provider: duckduckgo, bing, or searxng

Output:

{
  "items": [
    {
      "title": "Example Result",
      "url": "https://example.com",
      "snippet": "Result summary...",
      "source": "duckduckgo"
    }
  ],
  "providerUsed": "duckduckgo",
  "fallbackUsed": false,
  "triedProviders": ["duckduckgo"]
}

Fallback order:

• DuckDuckGo → SearXNG → Bing
• SearXNG → DuckDuckGo → Bing
• Bing → DuckDuckGo → SearXNG

fetch_url

Fetch a URL and return extracted content plus metadata in a normalized envelope.

Input:

{
  "url": "https://example.com/article",
  "format": "markdown",
  "max_length": 8000,
  "start_index": 0,
  "include_links": true,
  "include_media": true
}

Options:

Option	Description
url	URL to fetch
format	markdown, text, html, json, raw, or metadata
max_length	Maximum returned content characters, default 25000
start_index	Start content from this character index
engine	auto, http, or browser; browser fallback is reserved for future optional support
include_links	Include extracted links for HTML pages
include_media	Include extracted image/video/audio references for HTML pages
include_comments	Include comments for site adapters that support comments, default true for Reddit
comment_limit	Maximum comments for comment-capable adapters, max 100
comment_sort	top, best, new, or controversial
max_depth	Maximum comment nesting depth
timeout_ms	Request timeout override
fresh	Bypass in-memory cache
download	Save original fetched bytes to a managed local file and return it in attachments; default false
download_dir	Optional output directory for downloads; defaults to the system temp directory
download_ttl_seconds	Cleanup TTL for managed downloads, default 86400 seconds
max_download_bytes	Response/download byte cap override, additionally capped by MAX_BYTES

Output:

{
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "title": "Example Article",
  "content_type": "text/html",
  "resource_type": "html",
  "format": "markdown",
  "content": "# Example Article\n\n...",
  "metadata": {
    "status": 200,
    "content_type": "text/html",
    "byte_length": 12345,
    "extractor": "html",
    "fetched_at": "2026-05-03T00:00:00.000Z"
  },
  "links": [],
  "media": {
    "images": [],
    "videos": [],
    "audio": []
  },
  "truncated": false,
  "original_length": 1200,
  "start_index": 0,
  "next_start_index": null,
  "warnings": []
}

Supported Resources

Resource	Behavior
HTML pages	Extracts readable article content, title, metadata, optional links, and optional media references
Text and Markdown	Returns text directly with pagination support
JSON	Pretty-prints JSON when format is json or text-like when requested
XML and CSV-like text	Returns as text/data content
PDF	Extracts text and PDF metadata
Images	Returns metadata by default; saves the file only with download: true
Audio and video	Returns metadata by default; saves the file only with download: true
Archives and binary files	Returns metadata by default; downloads only when explicitly requested; archives are not auto-extracted
Reddit threads	Uses Reddit JSON endpoints and can include comments with limits

Local Downloads

fetch_url does not download binary/media files to disk by default. This avoids surprise disk usage and persistent local copies of arbitrary web content.

Use download: true when you need the original file available to another tool:

{
  "url": "https://httpbin.org/image/png",
  "format": "metadata",
  "download": true,
  "download_ttl_seconds": 86400
}

Download attachments look like this:

{
  "kind": "download",
  "path": "/tmp/mcp-web-search/downloads/mcp-fetch-id-image.png",
  "filename": "mcp-fetch-id-image.png",
  "original_filename": "image.png",
  "content_type": "image/png",
  "resource_type": "image",
  "byte_length": 8090,
  "sha256": "...",
  "expires_at": "2026-05-04T00:00:00.000Z"
}

Download safety behavior:

• Downloads are opt-in only.
• Files are written with 0600 permissions.
• Filenames are sanitized and prefixed with a managed artifact ID.
• SHA-256 is returned for verification.
• Expired managed artifacts are cleaned up through sidecar metadata.
• Cleanup only touches managed artifacts inside the configured download directory.
• Archives are never auto-extracted.

Reddit Thread Extraction

Reddit thread URLs are handled by a site adapter and fetched through Reddit JSON endpoints.

Input example:

{
  "url": "https://www.reddit.com/r/codex/comments/abc123/gpt55_is_so_good/",
  "include_comments": true,
  "comment_limit": 30,
  "comment_sort": "top",
  "max_depth": 2
}

The output uses resource_type: "site" and metadata.extractor: "reddit-thread".

Reddit public JSON can still rate-limit or return 403/429 depending on Reddit, subreddit rules, and request frequency. When that happens, retry later or reduce request frequency.

Providers

Provider	API Key Required	Notes
DuckDuckGo	No	Default, simple, no browser required
Bing	No	Uses Chrome/Chromium through Puppeteer
SearXNG	No	Best option for self-hosted high-volume usage

Environment Variables

Variable	Default	Description
DEFAULT_SEARCH_PROVIDER	duckduckgo	Default search provider: duckduckgo, bing, or searxng
SEARXNG_URL	http://localhost:8099	SearXNG instance URL
HTTP_TIMEOUT	15000	Request timeout in milliseconds
MAX_BYTES	20971520	Maximum fetched response/download size
MCP_COMPAT_MODE	unset	Set to legacy to simplify tools/list schemas for MCP clients with weak discovery parsers

SearXNG Setup

SearXNG is a free self-hosted meta-search engine.

Quick setup with Docker:

mkdir -p ~/docker/searxng

Create ~/docker/searxng/settings.yml with JSON enabled, then run the SearXNG container. The important setting is search.formats containing both html and json.

Example relevant setting:

search:
  formats:
    - html
    - json

Then set:

export SEARXNG_URL="http://localhost:8099"

Chrome Setup for Bing Provider

OS	Command
Ubuntu/Debian	sudo apt install chromium-browser
Fedora	sudo dnf install chromium
Arch	sudo pacman -S chromium
macOS	brew install --cask google-chrome

Custom path:

export CHROME_PATH="/path/to/chrome"

MCP Discovery Compatibility

Some MCP clients have weak schema parsers and fail during discovery on array-valued JSON Schema nodes such as enum or required.

If discovery fails, set:

export MCP_COMPAT_MODE="legacy"

This only simplifies advertised tools/list schemas. Tool execution behavior stays the same.

URL Safety

fetch_url blocks unsafe targets before fetching and before following redirects.

Blocked targets include:

• localhost hostnames
• .localhost and .local hostnames
• private IPv4 ranges
• IPv4 loopback, link-local, carrier-grade NAT, benchmark, multicast, and selected special-use ranges
• IPv4-mapped IPv6 addresses that resolve to blocked IPv4 ranges
• IPv6 loopback, unspecified, unique-local, multicast, and link-local ranges
• redirects that resolve to blocked addresses

The HTTP transport resolves and validates addresses before connecting, then connects to the vetted address while preserving the original host/SNI for normal HTTPS behavior.

Repository Structure

• src/server.ts - MCP server and tool schemas
• src/providers/ - search providers
• src/fetch/ - URL/resource loading pipeline
• src/fetch/content/ - shared content helpers such as Markdown conversion and readability fallback
• src/fetch/extractors/ - resource extractors for HTML, text/data, PDF, and media metadata
• src/fetch/site-adapters/ - domain-specific extractors such as Reddit threads
• src/utils/ - shared utilities
• test/ - Node test runner tests

Troubleshooting

Issue	Solution
Chrome not found	Install Chrome/Chromium or set CHROME_PATH
SearXNG 403	Enable JSON API in settings.yml
Timeout	Increase HTTP_TIMEOUT or pass timeout_ms
MCP discovery error: 'list' object has no attribute 'get'	Set MCP_COMPAT_MODE=legacy
Reddit 429 or 403	Reddit rate limited or blocked the JSON endpoint; retry later or reduce request frequency
Download missing from output	Set download: true; downloads are disabled by default
Download rejected as too large	Increase max_download_bytes within the server cap or raise MAX_BYTES

License

MIT