EliDev Scraper API

An asynchronous REST API, built using FastAPI, Redis, and the Ollama language model, enabling highly resilient web scraping across more than 12 major public platforms (Google Maps, Airbnb, Trustpilot, TripAdvisor, Booking.com, Instagram, YouTube, LinkedIn, TikTok, etc.).

---

Key Highlights

• Asynchronous Job Engine (202 Accepted Pattern): Immediate job acknowledgement with background extraction worker. Prevents gateway timeouts on heavy scraping jobs.
• Local AI Semantic Fallback (Ollama & Qwen2.5): Memory-optimized HTML text sanitizer coupled with local LLM extraction for unstructured or updated web layouts.
• Advanced Anti-Bot Evasion: Impersonates real browser TLS fingerprints via curl-cffi and employs headless Chromium with playwright-stealth for JS challenges.
• Smart Caching & URL Normalization: Automatically strips tracking query parameters (utm_*, fbclid, igsh) and resolves short URLs (maps.app.goo.gl, youtu.be). Caches results in Redis (12h TTL for success, 5m TTL for failures).
• Dynamic API Key & Credit Metering: Redis-backed API key storage with tier support (Free, Basic, Pro, Enterprise), credit balance metering per completed job, and HTTP 402 Payment Required enforcement.
• Webhook Callbacks: Push notifications sent via asynchronous HTTP POST to user-provided webhook_url endpoints upon job completion or failure.
• 1-Command Docker Deployment: Pre-configured docker-compose.yml bundling FastAPI, Redis, and host gateway bridging to host-running Ollama.

---

Platform Support

Category	Platform	Supported Endpoint	Scraped Data
Ratings	Google Maps	/api/v1/ratings	Rating (1-5), Total Review Count
	Trustpilot	/api/v1/ratings	TrustScore, Total Reviews
	TripAdvisor	/api/v1/ratings	Overall Rating, Review Count
	Airbnb	/api/v1/ratings	Star Rating, Review Count
	Google Play Store	/api/v1/ratings	App Rating, Total Downloads / Reviews
	Apple App Store	/api/v1/ratings	App Rating, Review Count
Reviews	Airbnb	/api/v1/reviews	Structured list of comments, author metadata, ratings, dates
	Microdata/JSON-LD	/api/v1/reviews	Standardized reviews extracted from structured data tags
Availability	Airbnb	/api/v1/availability	Nightly Price, Currency, Availability Status, Dates
	Booking.com	/api/v1/availability	Base Price, Currency, Next Available Date
	Abritel / VRBO	/api/v1/availability	Base Price, Currency, Availability Status
Social Proof	Instagram	/api/v1/social	Followers Count, Raw String Metric
	LinkedIn	/api/v1/social	Company / Profile Followers Count
	TikTok	/api/v1/social	Followers Count
	YouTube	/api/v1/social	Subscribers Count

---

Quick Start

Interactive API Documentation (Swagger)

Once the server is running, access the interactive Swagger UI at: http://127.0.0.1:8000/docs

---

Option 1: Docker Compose (Recommended)

Run the full stack (FastAPI + Redis + Ollama bridge) in seconds:

# 1. Clone the repository
git clone https://github.com/your-username/elidev-scraper-api.git
cd elidev-scraper-api

# 2. Copy the environment file
cp .env.template .env

# 3. Launch with Docker Compose
docker compose up -d --build

The API will be live at http://localhost:8000.

---

Option 2: Local Manual Setup

Prerequisites

• Python 3.11+
• Redis Server running locally on port 6379
• Ollama installed and running locally on port 11434

Steps

1. Clone the repository & create a virtual environment:

   git clone https://github.com/your-username/elidev-scraper-api.git
   cd elidev-scraper-api
   python -m venv .venv
   
   # Windows (PowerShell)
   .\.venv\Scripts\Activate.ps1
   # Linux/macOS
   source .venv/bin/activate

2. Install Python dependencies & Playwright Chromium:

   pip install -r requirements.txt
   playwright install chromium

3. Configure Environment Variables:

   cp .env.template .env

4. Start Redis Server:

   # Windows (via WSL or native Redis) / Linux / macOS
   redis-server

5. Start Uvicorn Web Server:

   uvicorn app.main:app --host 127.0.0.1 --port 8000 --reload

---

Setting Up Ollama (AI Fallback Engine)

This project uses Ollama to run local Large Language Models (default: qwen2.5:3b) as a fallback when standard HTML parsers encounter layout changes or complex structure.

1. Install Ollama: Download from https://ollama.com/
2. Pull the Qwen2.5 model:

   ollama pull qwen2.5:3b

3. Ensure Ollama is running:

   ollama serve

4. Docker Integration: docker-compose.yml automatically uses host.docker.internal:11434 to communicate with Ollama running on your host machine without needing to duplicate LLM weights inside containers.

---

Monetization & Key Administration Endpoints

Admin keys (role: admin) can manage client API keys and credit balances:

Endpoint	Method	Role	Description
/api/v1/auth/me	GET	All	Verify API key, inspect remaining credits, tier, and rate limit rules.
/api/v1/auth/keys	POST	Admin	Issue a new API key with custom owner, plan, credits, and rate limits.
/api/v1/auth/keys/{key}	GET	Admin	Inspect an existing API key's details and credits balance.
/api/v1/auth/keys/{key}/recharge	POST	Admin	Recharge an API key with additional credits.
/api/v1/auth/keys/{key}	DELETE	Admin	Revoke/deactivate an API key.

---

Webhook Notifications

Clients can pass an optional webhook_url query parameter on any scraping request:

curl -X GET "http://127.0.0.1:8000/api/v1/ratings?url=https://www.google.com/maps/place/Tour+Eiffel/data=!4m2!3m1!1s0x47e66e2964e34e2d:0x8ddca9ee380ef7e0&webhook_url=https://webhook.site/my-callback-endpoint" \
     -H "X-API-Key: pub_key_1"

When the extraction completes, the server sends an asynchronous HTTP POST request to the webhook_url:

Webhook Request Payload:

{
  "status": "completed",
  "url": "https://www.google.com/maps/place/Tour+Eiffel/data=!4m2!3m1!1s0x47e66e2964e34e2d:0x8ddca9ee380ef7e0",
  "data": {
    "rating": 4.7,
    "review_count": 1707654
  },
  "webhook_url": "https://webhook.site/my-callback-endpoint",
  "error": null,
  "updated_at": "2026-07-18T15:00:00.000000+00:00"
}

Webhook Headers:

• X-Scraper-Event: task.completed or task.failed
• X-Scraper-Timestamp: 2026-07-18T15:00:00Z

---

License

Distributed under the MIT License.

Developed by EliDev (https://eli-dev.fr/).