Clusy Crawler v0.2.0 Beta 3 Preview

Clusy Crawler v0.2.0 Beta 3 Preview

Fast, bounded, source-derived web extraction you can run on your own
infrastructure.

Beta 3 turns the optional quality path into a closed source-serialization
pipeline, hardens every runtime tier, and ships a release surface that is easy
to evaluate without a hosted account or model dependency.

Why try it

• Apache-2.0 and self-hosted; the deterministic path requires no model.
• Static, browser, and optional quality runtime profiles.
• Native Rust/PyO3 extraction with guarded HTTP/2 fetching and conditional
  Chromium rendering.
• Bounded URL, redirect, DOM, text, table, image, list, code, math, output,
  queue, and concurrency work.
• Source-derived quality selection with authenticated, replay-verified local
  serialization and deterministic fallback.
• Real container CI for authentication, SSRF rejection, browser sandboxing,
  configured quality extraction, dependency boundaries, and source identity.

Registered public result

On the separately registered Beta 2 AEB article_body tree (all 181 pages),
Clusy recorded F1 0.972127 versus 0.957546 for exact Trafilatura 2.1.0, a
paired delta of +0.014581 with a 95% bootstrap interval of
[+0.005547, +0.025336]. Machine-local in-memory extraction was
173.97 pages/s on the recorded machine.

That is a scoped, reproducible article-body comparison—not a universal web,
HTTP-service, or live-provider claim. The immutable receipt and retained
artifacts are in bench/evidence/.

Run it

git clone https://github.com/clusy-io/clusy-crawler.git
cd clusy-crawler
git checkout v0.2.0-beta.3
cp .env.example .env
docker compose up --build --detach
curl --fail http://127.0.0.1:11235/health/ready

Then send a URL to /crawl, /md, /html, or use /map for bounded
discovery. The default Compose service binds to loopback.

Release notes

See CHANGELOG.md
for the exact Added/Changed/Fixed boundaries and
docs/SELF_HOSTING.md
for production configuration.