Skip to content

v0.1.0 — Initial Open-Source Release

Choose a tag to compare

View all tags
@analavishal analavishal released this 18 Mar 07:25
· 23 commits to master since this release
v0.1.0
fb7e247

AnakinScraper OSS v0.1.0

The open-source web scraping API for AI. Turn any website into LLM-ready markdown or structured data.

Features

  • Sync + async scraping — POST /v1/scrape (blocks) or /v1/url-scraper (poll)
  • Batch scraping — up to 10 URLs in one request
  • Handler chain — fast HTTP fetch with automatic fallback to Camoufox anti-detect browser
  • Structured JSON extraction — Gemini 2.5 Flash AI extraction (bring your own API key)
  • Domain configs — per-domain handler selection, timeouts, retries, content validation, custom headers
  • Proxy auto-selection — Thompson Sampling picks the best proxy per domain
  • HTML to Markdown — intelligent content extraction with boilerplate removal

Quick Start

git clone https://github.com/AnakinAI/anakinscraper-oss.git && cd anakinscraper-oss && make up
curl -s -X POST http://localhost:8080/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq .markdown

CLI

pip install anakin-cli
anakin scrape "https://example.com" --api-url http://localhost:8080

Infrastructure

  • Single Go binary + Camoufox browser + PostgreSQL
  • Docker Compose for one-command deployment
  • 35 unit tests
  • AGPL-3.0 license
Assets 2
Loading