agent-webtool

Web fetch and multi-engine search tools for AI agents. No API keys required.

Exposes two tools through a single binary, usable as a CLI or as an MCP server:

Tool	Purpose
web_fetch	Fetch a URL and return its content as markdown, plain text, or raw HTML.
web_search	Query DuckDuckGo + Bing + Brave + Yahoo in parallel, deduplicate by URL, rank results via Reciprocal Rank Fusion (RRF, k=60).

Designed for agents that need first-class web access without depending on Google / Bing / SerpAPI accounts. Runs on Node.js ≥ 18 or Bun ≥ 1.0 — pick whichever you have.

---

Install

One-shot (no install, recommended for MCP)

# npm
npx -y agent-webtool fetch https://example.com
npx -y agent-webtool search "claude code mcp"

# bun
bunx agent-webtool fetch https://example.com
bunx agent-webtool search "claude code mcp"

Global install

# npm
npm install -g agent-webtool

# bun
bun add -g agent-webtool

# then either binary name works:
webtool fetch https://example.com
agent-webtool search "claude code mcp"

Both binary names (webtool and agent-webtool) point to the same CLI.

Project install

# npm
npm install --save-dev agent-webtool

# bun
bun add -d agent-webtool

# use in package.json scripts:
#   "search": "agent-webtool search ..."

Requirements: Node.js ≥ 18 (native fetch) or Bun ≥ 1.0.

---

Use as a CLI

Usage: webtool [options] [command]

Commands:
  fetch <url> [options]      Fetch a URL → markdown / text / html
  search <query> [options]   Multi-engine parallel search with RRF aggregation
  mcp [options]              Run an MCP stdio server exposing web_fetch / web_search

Run `webtool <command> --help` for per-command options.

Both commands print to stdout. When stdout is an interactive terminal, markdown is rendered with colors, headings, and clickable links (via marked-terminal). When stdout is a pipe / file / non-TTY, the same content is printed as raw markdown — perfect for > page.md or piping into another command. Use --raw to force raw output even in a terminal. NO_COLOR=1 also disables rendering.

webtool fetch

webtool fetch https://bun.sh                          # markdown, auto-rendered if TTY
webtool fetch https://example.com --format text       # plain text
webtool fetch https://example.com --format html       # raw HTML
webtool fetch https://example.com > page.md           # raw markdown to file
webtool fetch https://example.com --raw               # raw markdown in terminal

Options:

Flag	Default	Description
--format <fmt>	markdown	markdown | text | html
--max-bytes <n>	100000	Truncate output at this many bytes
--timeout-ms <n>	30000	Per-request timeout
--raw	—	Disable terminal markdown rendering (TTY only)

Rendering only applies to --format markdown. text and html are always printed verbatim.

webtool search

# All 4 engines in parallel
webtool search "bun javascript runtime" --limit 5

# Restrict to a subset
webtool search "typescript handbook" --engines brave,duckduckgo --limit 10

# Past-week news only
webtool search "ai breakthroughs" --time week

# Site-scoped
webtool search "structured outputs" --site docs.anthropic.com

Options:

Flag	Default	Description
--engines <list>	duckduckgo,bing,brave,yahoo	Comma-separated subset
--limit <n>	10	Max aggregated results (1–30)
--time <range>	—	day | week | month | year (engines may ignore)
--site <domain>	—	Restrict to a domain (injects site: operator)
--raw	—	Disable terminal markdown rendering (TTY only)

Exit codes

Code	Meaning
0	Success
1	Generic error
2	Invalid input (URL, engine name, schema validation)
3	Network failure (or all engines failed in search)

---

Use as an MCP server

agent-webtool ships an MCP server over stdio. Any MCP-compatible client can connect.

# npm
npx -y agent-webtool mcp                      # both tools
npx -y agent-webtool mcp --tools fetch        # only web_fetch
npx -y agent-webtool mcp --tools fetch,search # both, explicit

# bun
bunx agent-webtool mcp
bunx agent-webtool mcp --tools fetch,search

The integrations below use npx -y in their examples. If you prefer Bun, replace npx -y with bunx and npx (the launcher in the args array) with bunx.

Claude Code

# User scope (available in every project; recommended)
claude mcp add --scope user webtool -- npx -y agent-webtool mcp

# Project scope (writes ./.mcp.json, shared with teammates via git)
claude mcp add --scope project webtool -- npx -y agent-webtool mcp

# Bun-based equivalent
claude mcp add --scope user webtool -- bunx agent-webtool mcp

# Verify
claude mcp list

Codex CLI

codex mcp add webtool -- npx -y agent-webtool mcp

# Bun-based equivalent
codex mcp add webtool -- bunx agent-webtool mcp

# Verify
codex mcp list

The entry is written to ~/.codex/config.toml under [mcp_servers.webtool].

Claude Desktop

No CLI for adding servers — edit the config file directly. On macOS: ~/Library/Application Support/Claude/claude_desktop_config.json.

{
  "mcpServers": {
    "webtool": {
      "command": "npx",
      "args": ["-y", "agent-webtool", "mcp"]
    }
  }
}

Cursor

Edit ~/.cursor/mcp.json (user) or .cursor/mcp.json (project):

{
  "mcpServers": {
    "webtool": {
      "command": "npx",
      "args": ["-y", "agent-webtool", "mcp"]
    }
  }
}

Continue / any generic MCP client

Spawn npx -y agent-webtool mcp and speak the MCP protocol over stdio. The server advertises two tools (web_fetch, web_search) and their JSON Schemas via tools/list.

---

Output format

Both tools return plain text — no JSON wrapping, no metadata envelope. Pipe it straight into a file or another command.

web_fetch

Returns the page content as a string in the requested format (markdown / text / html). For example, fetching https://example.com in markdown mode prints:

Example Domain

# Example Domain

This domain is for use in documentation examples without needing permission. Avoid use in operations.

[Learn more](https://iana.org/domains/example)

When a URL redirects to a different host, the output is a single line you can act on:

[Redirected to a different host: https://final-host.example/]
[Call web_fetch again with the redirect URL to follow.]

If the content exceeds --max-bytes, the output ends with a [truncated] marker.

web_search

Returns a numbered markdown list, one entry per line pair (title link + snippet):

1. [Bun — A fast all-in-one JavaScript runtime](https://bun.sh/)
   Bundle, install, and run JavaScript & TypeScript — all in Bun.

2. [GitHub - oven-sh/bun: Incredibly fast JavaScript runtime, bundler, test runner, and package manager](https://github.com/oven-sh/bun)
   Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one.

3. [Bun (software) - Wikipedia](https://en.wikipedia.org/wiki/Bun_(software))
   Bun is a JavaScript runtime, package manager and test runner designed as a drop-in replacement for Node.js.

If some engines fail (timeout / challenge page / parse error), or return a page with zero parsed hits, footer lines appear at the end:

> Note: 1 engine(s) failed — duckduckgo.
> Note: 1 engine(s) returned no results — brave.

If all engines fail, the CLI exits with code 3 and prints the error to stderr.

Aggregation: results from each engine are pulled in parallel, URLs are normalized (HTTPS-upgraded, www. stripped, tracking params removed, trailing slash trimmed, query keys sorted), then merged across engines. Final ranking uses Reciprocal Rank Fusion (score = Σ 1 / (60 + rank)).

---

Behavior & security

• HTTPS upgrade. http:// URLs are auto-upgraded to https://.
• Same-origin redirects only. Up to 10 hops, host compared modulo a leading www.. Cross-origin redirects are reported in the output (not followed) — call again with the new URL to follow.
• SSRF guard. Private, loopback, and link-local addresses (RFC 1918, 127/8, 169.254/16, IPv6 ULA/link-local, ::ffff: mapped privates) are rejected. Set WEBTOOL_ALLOW_PRIVATE=1 to allow localhost for development.
• Hard 10 MB cap on fetched response body.
• 15-minute LRU cache on web_fetch (keyed by URL + format + maxBytes; 256 entries / 50 MB cap).
• Per-engine 15s timeout in web_search. Engines that fail or return zero parsed hits are reported in the footer; others still return results (partial success). If every engine fails, the call errors.
• No telemetry. No third-party API keys. All requests go directly to the target host.

---

Library use (optional)

This package primarily ships a CLI / MCP binary. If you want to call the core functions from JavaScript, install the source and import from src/:

import { webFetch, webSearch } from 'agent-webtool/src/index.ts' // requires bun or a TS-aware loader

const markdown = await webFetch({ url: 'https://example.com' })  // → string
const list     = await webSearch({ query: 'bun runtime' })       // → string (markdown list)

Both functions return a plain string. A dedicated library entry (exports-mapped, with .d.ts) may be added in a future release.

---

Development

git clone https://github.com/potato47/agent-webtool.git
cd agent-webtool
bun install
bun test            # 51 fixture-based tests; no network
bun run cli -- search "test" --limit 3
bun run build       # produces dist/cli.mjs (single ESM bundle)

The build is a single self-contained ESM file that runs under plain Node ≥ 18. Bun is only required at dev time.

Refreshing engine selectors when a SERP changes:

bun scratch/probe-engines.ts          # captures fresh HTML to scratch/dump/
bun scratch/peek.ts                   # tests parsers against fresh capture
# then update selectors in src/core/engines/*.ts

---

Contributing

Issues and pull requests welcome at https://github.com/potato47/agent-webtool.

When opening a bug report, please include:

• the command you ran (or MCP tools/call request),
• the full output (use --raw for searches so the markdown is verbatim),
• your Node / Bun version (node -v, bun -v) and OS.

---

License

MIT — see LICENSE.