CrawlScope

Safe web monitoring and change intelligence for operational teams.

CrawlScope turns recurring page checks into an evidence-backed workflow. It retrieves a bounded public resource, extracts only the relevant content, fingerprints each observation, calculates a normalized change score, and routes material differences to a human review queue.

Why it is more than a scraper

• SSRF protection blocks localhost, private, reserved, and unresolved network targets.
• Timeouts, redirect limits, selector boundaries, and a response-size ceiling constrain collection.
• SHA-256 fingerprints make unchanged observations cheap and explicit.
• Before-and-after evidence is stored with a severity score and review state.
• Review actions are idempotent and the CSV export has a stable downstream schema.
• Deterministic mock:// sources provide a reliable demo without scraping third-party websites.

Capabilities

• Target inventory with cadence, selector, threshold, and health metadata
• On-demand observations and production-ready worker boundary
• HTML-to-text normalization and CSS selector extraction
• Content hashing and similarity-based change scoring
• Human review queue with acknowledge/dismiss actions
• Run history, latency, response codes, evidence, and CSV export
• Responsive operator dashboard and OpenAPI documentation
• Automated API tests and GitHub Actions workflow

Quick start

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt
.\.venv\Scripts\python.exe -m uvicorn app.main:app --reload --port 8001

Open http://127.0.0.1:8001. API documentation is available at /docs.

Or run the containerized demo:

docker compose up --build

Verification

.\.venv\Scripts\python.exe -m ruff check app tests
.\.venv\Scripts\python.exe -m pytest

Architecture

Target registry -> URL safety gate -> bounded fetch/extract -> normalized fingerprint
                                                           |
Previous observation -> similarity comparison -------------+
                                                           v
                                      immutable run + change evidence
                                                           |
                                                           v
                                            human review + CSV export

See docs/ARCHITECTURE.md for trust boundaries and production evolution.

Demo boundaries

The local MVP executes runs on demand and stores data in SQLite. Production scheduling, distributed workers, object-storage snapshots, tenant authentication, and outbound notification delivery are documented extension points rather than simulated guarantees. All demo targets and observations are synthetic.

License

MIT