Skip to content

xorsoftdev/webscape

Repository files navigation

webscape

🚧 This project is under active development and not ready for production use. Expect breaking changes.

webscape is a metasearch engine written in Rust 🦀.

The project consists of two crates:

  • webscape - metasearch library inspired by SearXNG.
  • webscape-server - MCP server armed with search and browse tools that acts as a web gateway for LLMs.

Install

Download webscape-server from GitHub Releases. Support for more platforms coming soon.

Alternatively, you can build from source. See CONTRIBUTING.md.

Why the project exists

I needed a lightweight MCP server to extend a local LLM with search capabilities. While SearXNG remains the go-to option, it requires Docker and a third-party MCP server. webscape, on the other hand, handles this use case with a single binary.

Key differences:

  • Written in Rust.
  • Ships a zero-dependency MCP server binary.
  • Browser emulation via wreq-util, which helps avoid bot detection based on TLS (JA3/JA4) and HTTP/2 fingerprints.
  • Unlike SearXNG, it doesn't yet provide a web interface or an API.

Providers

Name Alias Status Description
Google google unstable Emulates an Android app user agent.
Bing bing stable Scrapes the Bing search page.
DuckDuckGo duckduckgo or ddg stable Scrapes the DuckDuckGo HTML page.
Apple Maps apple_maps stable Queries an Apple MapKit API endpoint.

MCP server

MCP server is a lightweight HTTP server that exposes MCP endpoint at http://<host>:<port>/mcp using Streamable HTTP.

webscape-server can be integrated with popular LLM clients:

See Integrations for detailed instructions.

Quick start

⚠️ MCP endpoint is unauthenticated and browse tool fetches any given URL. Run it on 127.0.0.1 (default) and do not expose it to untrusted networks. Binding to 0.0.0.0 exposes an open web-fetch proxy (SSRF risk).

To start the server, run webscape-server.

# start the server and listen on 127.0.0.1:8888
webscape-server
# custom config file
webscape-server --config config.toml
# run with more verbosity
RUST_LOG=webscape_server=debug webscape-server

MCP tools

MCP server provides the following tools:

  • search — search the internet with Google, Bing, DuckDuckGo, or Apple Maps.
  • browse — browse web pages and extract their content.

search tool

Field Type Default Description
query string - Search query (required).
provider enum duckduckgo google, bing, duckduckgo, apple_maps

If provider is omitted, the default from config is used (see Configuration).

browse tool

Fetches one or more web pages and extracts their content using rs-trafilatura. Returns one text block per URL (URL, title, and content).

Field Type Default Description
urls string[] - Web page URLs (required)

Configuration

webscape-server supports basic configuration. All parameters are optional.

Default config:

[defaults]
provider = "duckduckgo"
fallback = ["google", "bing", "duckduckgo"]

See [config.example.toml](config.example.toml) for the full reference.

License

AGPL-3.0

Packages

 
 
 

Contributors

Languages