Turn any URL into clean, structured, LLM-ready content.
The open-source web fetching & content extraction API.
SparkFetch is a powerful, self-hostable API that crawls and extracts web content — converting messy HTML into clean Markdown, structured JSON, or plain text. Built for AI applications, RAG pipelines, research tools, and any workflow that needs reliable web data.
Key capabilities:
- 🌐 Scrape — Fetch any URL and get clean Markdown + metadata
- 🕷️ Crawl — Recursively crawl entire websites with depth control
- 🗺️ Map — Discover all URLs on a domain instantly
- 🧹 Clean output — Strip nav, ads, and boilerplate automatically
- 📦 Structured data — Returns JSON with title, description, links, and content
- ⚡ Fast — Built on Node.js 24 + Express 5
- Node.js 18+
- pnpm 9+
# Clone the repository
git clone https://github.com/Sparkfetch/sparkfetch.git
cd sparkfetch
# Install dependencies
pnpm install
# Start the development server
pnpm --filter @workspace/api-server run devThe API will be available at http://localhost:5000/api.
Fetch a single URL and return its content as Markdown with metadata.
Request:
{
"url": "https://example.com",
"formats": ["markdown", "html"],
"includeTags": ["article", "main"],
"excludeTags": ["nav", "footer", "aside"]
}Response:
{
"success": true,
"data": {
"markdown": "# Page Title\n\nContent here...",
"metadata": {
"title": "Page Title",
"description": "Page description",
"url": "https://example.com",
"statusCode": 200,
"fetchedAt": "2025-01-01T00:00:00Z"
}
}
}Crawl a website recursively and extract content from all pages.
Request:
{
"url": "https://example.com",
"maxDepth": 2,
"limit": 10,
"excludePaths": ["/login", "/admin"]
}Response:
{
"success": true,
"jobId": "crawl_abc123",
"status": "queued",
"message": "Crawl job started. Use GET /api/v1/crawl/:jobId to check status."
}Check the status of a crawl job.
Response:
{
"success": true,
"status": "completed",
"completed": 8,
"total": 8,
"data": [
{
"url": "https://example.com",
"markdown": "# Home\n\n...",
"metadata": { "title": "Home", "statusCode": 200 }
}
]
}Discover all accessible URLs on a domain.
Request:
{
"url": "https://example.com",
"limit": 100
}Response:
{
"success": true,
"links": [
"https://example.com/",
"https://example.com/about",
"https://example.com/blog"
]
}Health check endpoint.
| Use Case | How SparkFetch Helps |
|---|---|
| AI / RAG pipelines | Feed clean Markdown into your LLM context |
| Research automation | Batch-crawl sources and extract structured info |
| Content monitoring | Track page changes with periodic scraping |
| Data collection | Turn any website into structured JSON datasets |
| Documentation indexing | Crawl and index docs sites for search |
sparkfetch/
├── artifacts/
│ └── api-server/ # Express API (TypeScript)
│ └── src/
│ ├── routes/
│ │ └── v1/ # Versioned API routes
│ └── lib/ # Utilities (fetcher, extractor, markdown)
├── lib/
│ ├── api-spec/ # OpenAPI 3.1 spec + codegen
│ ├── api-zod/ # Zod validation schemas (generated)
│ └── db/ # Drizzle ORM schema
└── .github/
└── workflows/
└── ci.yml # GitHub Actions CI
# Production build
pnpm --filter @workspace/api-server run build
# Set environment variables
export DATABASE_URL="postgresql://..."
export PORT=5000
# Start
pnpm --filter @workspace/api-server run startContributions are welcome! Please open an issue first to discuss what you'd like to change.
- Fork the repository
- Create your feature branch (
git checkout -b feat/amazing-feature) - Commit your changes (
git commit -m 'feat: add amazing feature') - Push to the branch (
git push origin feat/amazing-feature) - Open a Pull Request
MIT © SparkFetch