v0.8.0 — P2P Cache + API Detector

What's New in v0.8.0

🌐 P2P Decentralized Cache Network

• WebSocket server+client — each PC is a peer
• DHT-lite lookup to top-5 peers by reputation
• Gossip protocol — broadcast updates to random 3 peers
• Privacy: PII stripping, embeddings-only sharing, no prompts leaked
• Graceful degradation: P2P down → works locally, no crashes
• Peer persistence via ~/.harvest/p2p_peers.json
• Auto-port selection (8765-8774), auto-disable after 10 errors
• CLI: p2p-stats, p2p-peers, p2p-enable, p2p-disable
• 24 tests, all passing

🔍 API Reverse Engineering Agent

• harvest detect-api — discover hidden REST/GraphQL APIs
• Live network interception via patchright
• httpx/curl code generation, JSON export
• Tested on Twitch (GraphQL), GitHub (REST), 50+ sites

Improvements

• README rewritten (bilingual RU/EN)
• CHANGELOG.md created
• 139 tests total, all passing

Full Changelog: https://github.com/zad111ak-ai/harvest/blob/main/CHANGELOG.md

v0.6.3 — Self-Healing Scrapers, Semantic Cache, MCP Server

What's New

Self-Healing Parsers — When a website changes its HTML, Harvest automatically regenerates CSS selectors via LLM. No manual fixes needed.

Semantic Cache — Caches by meaning, not exact URL. Ask "get prices" then "extract product prices": second call is instant, 0 tokens. 50-70% token reduction on real workloads.

Structural Diff — Git diff for web pages. See what changed, what moved, what's new. Recommendations included.

Script Generator — Analyze page once, generate standalone Python script. Zero tokens at runtime.

MCP Server — Works with Claude, Cursor, Hermes, custom agents out of the box.

Honest Limitations

• Turnstile checkbox (behavioral biometrics) can still block
• Requires Chromium (~300MB)
• LLM extraction needs an endpoint (Ollama, OmniRoute, or OpenAI-compatible)
• Brand new — no community yet

Quick Start

pip install harvest-agent
# or
git clone https://github.com/zad111ak-ai/harvest
cd harvest && pip install -e .

Links

• PyPI: https://pypi.org/project/harvest-agent/
• Dev.to: https://dev.to/joseph_postman_b091ffaccf/i-built-a-web-scraper-that-fixes-itself-when-websites-change-then-i-open-sourced-it-4jgd

MIT Licensed. Built in 3 weeks.

v0.5.0

v0.5.0 — LLM Extraction + Enhanced Stealth

New Features

• LLM extraction — describe what you want in natural language, get structured JSON. No CSS selectors needed
• Enhanced stealth — 24 rotating User-Agents (Chrome/Firefox/Edge/Safari), randomized viewport/timezone/locale/platform
• Response caching — in-memory TTL cache (5 min default), zero-cost repeat requests
• Rate limiting — token bucket (10 req/min default), configurable
• Persistent browser session — one session reused across requests
• Adaptive error logging — self-learning loop integration

Fixes

• CaptchaSolver import fix
• Removed old GIF artifacts

Package

• PyPI package: harvest-agent (CLI commands unchanged: harvest, harvest-mcp)
• Install: pip install harvest-agent or git clone

Breaking Changes

• Version bumped to 0.5.0 (SemVer x.5 cadence: 0.5.5, 0.6.0, 0.6.5...)

Tests

• 75/75 passed (14 new tests for cache, rate_limiter, stealth)

v0.4.0 — MCP Server & Driftz

What's new

MCP Server (harvest-mcp)

Harvest now speaks the Model Context Protocol — the standard interface for AI agents. Connect it to Hermes, Claude Code, Cursor, or any MCP-compatible client.

7 tools available:

• — extract page content
• — structured CSS extraction
• — concurrent URL processing
• — collect emails/social links
• — full site crawling
• — change detection with diffs
• — system configuration

Driftz temp email

New module for temporary email addresses via driftz.net, mail.tm, and guerrillamail.com.

Refactored internals

• BrowserSession → Scrapling AsyncStealthySession
• Config manager with YAML/JSON support
• Target manager with automatic dead target cleanup
• Pipeline chaining across all operations
• ruff formatting, explicit re-exports

Quality

• 61 tests (all passing)
• ruff check clean
• ruff format clean