Repository navigation
First public release of Labyrinth — a PHP 8.1+ drop-in that defends a site against content scrapers.
Four layers, applied in order on every request:
- Verified-crawler allowlist — Googlebot / Bingbot / Yandex are confirmed by forward-confirmed reverse DNS (PTR ends in an official crawler domain and the hostname resolves back to the client IP), with an optional static CIDR safety net you maintain. Verified crawlers bypass everything, so you never risk deindexing.
- Scraper heuristics — weak, combinable signals (impossible UA version, no PTR, missing
Sec-Fetch-*/Sec-CH-UA, deep hit with no referer, fetches HTML but never assets, stdlib-defaultAccept-Language) are scored against a configurable threshold. Flags, never hard-blocks. - Tarpit labyrinth — flagged clients get a slow
noindexpage of endless fake links, a fake paginator, and link-only bait paths. No destructive payloads — only slow junk HTML. Routing is by client, so every path leads back into the maze. - Volume caps — per-IP request-rate and daily-download limits behind a pluggable store. This is the real ceiling against clients the heuristics cannot catch.
Includes
- Framework-agnostic
Labyrinth::handle(Request): ?Responseplus a PSR-15 middleware adapter. - Pluggable
StoreInterface:FileStore(default),ApcuStore,RedisStore,InMemoryStore. - Front-controller example and a full config reference.
- 52 PHPUnit tests; CI green on PHP 8.1, 8.2 and 8.3.
Honest about limits
A careful headless browser on residential proxies defeats the heuristics; the volume caps are what actually bound it. Heuristic UA bounds and static crawler ranges need periodic maintenance. See the README threat model.
Security hardening (included)
- Asset-extension allowlist trimmed to true static assets only — data formats (json/webmanifest) no longer let a scraper append
.jsonto a content path and skip scoring. FileStorerequires an explicit, app-owned private directory (no temp fallback), creates it mode0700, refuses a group/world-accessible or non-owned directory, and refuses symlinked state files.