Skip to content

v0.1.0

Latest

Choose a tag to compare

@North-web-dev North-web-dev released this 25 Oct 16:17
· 1 commit to main since this release

First public release of Labyrinth — a PHP 8.1+ drop-in that defends a site against content scrapers.

Four layers, applied in order on every request:

  1. Verified-crawler allowlist — Googlebot / Bingbot / Yandex are confirmed by forward-confirmed reverse DNS (PTR ends in an official crawler domain and the hostname resolves back to the client IP), with an optional static CIDR safety net you maintain. Verified crawlers bypass everything, so you never risk deindexing.
  2. Scraper heuristics — weak, combinable signals (impossible UA version, no PTR, missing Sec-Fetch-*/Sec-CH-UA, deep hit with no referer, fetches HTML but never assets, stdlib-default Accept-Language) are scored against a configurable threshold. Flags, never hard-blocks.
  3. Tarpit labyrinth — flagged clients get a slow noindex page of endless fake links, a fake paginator, and link-only bait paths. No destructive payloads — only slow junk HTML. Routing is by client, so every path leads back into the maze.
  4. Volume caps — per-IP request-rate and daily-download limits behind a pluggable store. This is the real ceiling against clients the heuristics cannot catch.

Includes

  • Framework-agnostic Labyrinth::handle(Request): ?Response plus a PSR-15 middleware adapter.
  • Pluggable StoreInterface: FileStore (default), ApcuStore, RedisStore, InMemoryStore.
  • Front-controller example and a full config reference.
  • 52 PHPUnit tests; CI green on PHP 8.1, 8.2 and 8.3.

Honest about limits

A careful headless browser on residential proxies defeats the heuristics; the volume caps are what actually bound it. Heuristic UA bounds and static crawler ranges need periodic maintenance. See the README threat model.

Security hardening (included)

  • Asset-extension allowlist trimmed to true static assets only — data formats (json/webmanifest) no longer let a scraper append .json to a content path and skip scoring.
  • FileStore requires an explicit, app-owned private directory (no temp fallback), creates it mode 0700, refuses a group/world-accessible or non-owned directory, and refuses symlinked state files.