ShadowCrawl

Async OSINT / multi-source scraping engine with dual-path fetching (aiohttp + Playwright), Redis queues, MongoDB persistence, proxy rotation, and anti-bot heuristics.

Ethical use only: scrape public pages you are allowed to access. Respect robots.txt, ToS, and local law.

---

Project Pitch

ShadowCrawl is a portfolio-grade collector for public-source monitoring (career pages, news, brand signals). It shows interviewers you can design resilient async workers, choose the right fetch path for static vs JS-heavy sites, and persist both raw and normalized OSINT documents idempotently.

---

Architecture Overview

┌──────────────────┐   LPUSH    ┌────────────────────┐
│  CLI / scheduler │ ─────────▶ │  Redis job queue   │
└──────────────────┘            │  (+ DLQ + stats)   │
                                └─────────┬──────────┘
                                          │ BRPOP
                     ┌────────────────────▼────────────────────┐
                     │           ScrapeOrchestrator            │
                     │ rate-limit → fetch → parse → upsert     │
                     └───────┬──────────────────────┬──────────┘
                             │                      │
                ┌────────────▼──────────┐ ┌─────────▼────────────┐
                │ aiohttp worker pool   │ │ Playwright pool      │
                │ + proxy + antibot     │ │ (JS-rendered pages)  │
                └────────────┬──────────┘ └─────────┬────────────┘
                             └──────────┬───────────┘
                                        │
                             ┌──────────▼──────────┐
                             │       MongoDB       │
                             │ raw_pages/documents │
                             └─────────────────────┘

AUTO strategy: try aiohttp first → if a JS shell is detected, escalate to Playwright.

---

Tech Stack Grid

Area	Tools
Runtime	Python 3.12, asyncio
Fetch	aiohttp, Playwright (Chromium), BeautifulSoup4 / lxml
Queue	Redis lists (queue + DLQ + counters)
Storage	MongoDB (raw HTML + normalized docs)
Validation	Pydantic v2
Ops	Docker Compose, Structlog, Typer CLI
Tests	pytest-asyncio + in-memory fakes

---

Quick Start (1 command)

cp .env.example .env && docker compose up --build -d

Enqueue demo jobs and inspect metrics:

docker compose run --rm cli demo
docker compose run --rm cli stats
docker compose run --rm cli show https://example.com/

Local (optional)

python -m venv .venv && source .venv/bin/activate
make install
docker compose up -d redis mongo
cp config/local.env .env
shadowcrawl demo
shadowcrawl worker --mode all

---

Key Engineering Challenges Solved

1. Dual-path fetching — static pages stay cheap (aiohttp); JS apps use Playwright without paying Chromium cost everywhere.
2. Backpressure & fairness — per-domain Redis rate limits + asyncio semaphores.
3. Anti-bot hygiene — UA rotation, human-like delays, captcha/block heuristics, proxy rotation hooks.
4. Idempotent OSINT storage — SHA-256 content hashing prevents noisy rewrites in MongoDB.
5. Failure isolation — exponential backoff retries and a dead-letter queue for poison jobs.
6. Operator UX — Typer CLI for enqueue / worker / stats that mirrors real scraping ops.

---

CLI Snapshot

Command	Description
shadowcrawl enqueue URL [-s auto|aiohttp|playwright]	Queue one job
shadowcrawl enqueue-file PATH	Bulk enqueue
shadowcrawl worker [--mode all|aiohttp|playwright]	Consume queue
shadowcrawl demo	Public sample targets
shadowcrawl stats	Queue + Mongo + counters

---

Key OSINT Metrics

Metric	Meaning
queue_depth / dlq_depth	Pending vs permanently failed
succeeded / failed / retried	Throughput & resilience
unchanged_hash	Idempotent skips
rate_limited / blocked	Throttle & antibot hits

---

License

MIT