RustyWeb

RustyWeb is a standalone crawler and graph-shaped search engine. Give it seed URLs, deploy it on Railway, and it crawls pages into a local persisted corpus that can be searched through a SERP or JSON API.

This repo is intentionally smaller than RustyRed. It does not ship the RustyRed graph database, MCP server, gRPC server, versioned graph compiler, or Web Commons federation merge protocol. It keeps only the crawler/search product surface.

Run Locally

cargo run

Open http://127.0.0.1:8080.

Seed a crawl:

curl -X POST http://127.0.0.1:8080/crawl \
  -H 'content-type: application/json' \
  -d '{"seeds":["https://example.com"],"max_pages":20,"max_depth":1}'

Search:

curl 'http://127.0.0.1:8080/search.json?q=example'

Railway

1. Create a new Railway service from this repo.
2. Add a volume mounted at /data.
3. Set RUSTYWEB_SEEDS to comma-separated URLs.
4. Optional but recommended: set RUSTYWEB_WRITE_TOKEN and send it as Authorization: Bearer ... for POST /crawl.

Useful environment variables:

Variable	Default	Purpose
RUSTYWEB_BIND	0.0.0.0:$PORT or 0.0.0.0:8080	Server bind address.
RUSTYWEB_DATA_DIR	./data	Corpus persistence directory. Use /data on Railway volume.
RUSTYWEB_SEEDS	unset	Comma-separated boot crawl seeds.
RUSTYWEB_MAX_PAGES	40	Default crawl page budget.
RUSTYWEB_MAX_DEPTH	1	Default crawl link depth.
RUSTYWEB_FOLLOW_OFFSITE	false	Whether crawls can leave seed domains.
RUSTYWEB_WRITE_TOKEN	unset	If set, protects POST /crawl.
RUSTYWEB_USER_AGENT	RustyWeb/0.1 ...	User agent for page and robots requests.

HTTP API

Route	Method	Description
/	GET	Search UI.
/search?q=	GET	Search UI with results.
/search.json?q=	GET	JSON search payload.
/crawl	POST	Start a background crawl job.
/crawl/:run_id	GET	Read crawl job status.
/health	GET	Process health.
/ready	GET	Corpus loaded and writable.

Search Model

RustyWeb stores pages and links. Search scores lexical matches in title, URL, and body, then returns the link neighborhood around the strongest hits so the result feels like a small map of the crawled web rather than only a list.

The crawler respects robots.txt when available, blocks localhost/private-network URLs, applies conservative budgets, and follows only seed domains unless follow_offsite is enabled.