Spider (spider-cloud)

Spider is a Rust-based, AI-friendly web scraping and crawling cloud. Point it at a URL and get back clean markdown, structured JSON, screenshots, or links — at up to 100K pages per second — with anti-bot bypass, residential proxies, headless browsers, and native MCP, LangChain, LlamaIndex, CrewAI, and AutoGen integrations.

URL: Visit APIs.json URL

Scope

• Type: Contract
• Position: Consuming
• Access: 3rd-Party

Tags

• Crawling, Scraping, Data Extraction, URLs, AI, Markdown, MCP, Rust, Headless Browser, Proxies

Timestamps

• Created: 2026-05-25
• Modified: 2026-05-25

APIs

Spider API

The Spider Cloud REST API exposes core /crawl, /scrape, /search, /links, /screenshot, /unblocker, and /transform endpoints, plus AI-discovered per-domain /fetch endpoints, residential/ISP/mobile proxy routing via proxy.spider.cloud, CDP/WebDriver BiDi cloud browsers, scraper directory lookups, crawl logs, and a credits endpoint. All endpoints accept and return JSON (also XML, CSV, JSONL via content-type) and authenticate with Bearer API keys. Up to 10,000 RPM per account.

Human URL: https://spider.cloud/docs/api

Base URL: https://api.spider.cloud

Tags

• Crawling, Scraping, Data Extraction, URLs, AI, Markdown, Search, Screenshot, Headless Browser, Proxies

Properties

• Documentation
• Documentation
• OpenAPI
• Authentication
• RateLimits
• NaftikoCapability — Crawling
• NaftikoCapability — Scraping
• NaftikoCapability — Search and Links
• NaftikoCapability — Browser and Screenshot
• NaftikoCapability — Account Data

Spider Cloud MCP Server

Spider's hosted Model Context Protocol server exposes 22 tools — eight core operations (crawl, scrape, search, links, screenshot, unblocker, transform, get_credits), five AI-routed variants, and nine browser-automation tools — to any MCP-compatible client. Authenticated with a Bearer API key. Also runnable locally via npx spider-cloud-mcp.

Human URL: https://spider.cloud/docs/integrations/mcp

Base URL: https://mcp.spider.cloud/mcp

Tags

• MCP, AI, Agents, Crawling, Scraping, Browser Automation

Properties

• Documentation
• SourceCode
• Authentication

Common Properties

• Website
• Documentation
• API Reference
• Guides
• Pricing
• Status
• GitHub Organization
• Source — Spider Core (Rust)
• SDKs (clients)
• SDK — Python
• SDK — Node.js
• SDK — Rust
• MCP Endpoint
• Integration — MCP
• Integration — LangChain
• Integration — LlamaIndex
• Integration — CrewAI
• Integration — FlowiseAI
• Integration — Agno
• Integration — Zapier
• Integration — x402
• Support
• Plans
• Rate Limits
• FinOps

Generated Artifacts

• openapi/ — OpenAPI 3.0.3 specification covering the documented Spider Cloud REST endpoints (/crawl, /scrape, /search, /links, /screenshot, /unblocker, /transform, /fetch/{domain}/{path}, /data/scraper-directory, /data/crawl_logs, /data/credits).
• capabilities/ — Five Naftiko capability files grouping the API surface into Crawling, Scraping, Search and Links, Browser and Screenshot, and Account Data. Each capability exposes both REST and MCP adapters.
• plans/ — API Commons Plans 0.1 file mapping Spider's pay-as-you-go credit model: $1/GB bandwidth + $0.001/compute-minute, with free sign-up credits, an AI-routing subscription tier, and enterprise custom pricing.
• rate-limits/ — API Commons Rate Limits 0.1 file documenting the default 10,000 RPM per-account limit, browser session concurrency (5) and idle timeout (5 min), and enterprise-negotiated overrides.
• finops/ — FOCUS 1.3 / FinOps Framework mapping with bandwidth, compute, successful-request, and credit-balance meters plus unit-economics targets (cost per GB, cost per compute minute, cost per 1K successful requests).

Maintainers

FN: Kin Lane

Email: kin@apievangelist.com