Clusy Crawler v0.2.0 Beta 4 Preview

Clusy Crawler v0.2.0 Beta 4 Preview

Fast, bounded, source-derived web extraction that you can run on your own infrastructure and inspect end to end.

What ships

• Static, browser, and optional quality images with explicit dependency boundaries.
• Guarded HTTP/2 fetching, redirect-by-redirect SSRF checks, and conditional sandboxed Chromium.
• Native Rust/PyO3 extraction, bounded local fallbacks, source-family specialists, deterministic discovery, and provenance-rich output.
• Correct production token, Chromium sandbox, benchmark-replay, and archive-availability documentation.
• The official Clusy wordmark, launch card, poster, and an 11.8-second X-ready animation.

Engine lineage

The generic HTML selector is Trafilatura-family software, not a from-scratch Clusy selector. The native path uses vendored rs-trafilatura broad 0.2.2 for the general candidate and a 9261e08-derived article backend for explicit article_body plus bounded adaptive rescue. The local fallback set includes Python Trafilatura 2.1.0, Readability, Markdownify, documentation-specific extraction, and raw-text rescue.

Clusy adds the guarded service, routing, specialists, candidate admission and comparison, structure recovery, Document IR and receipt primitives, crawl budgets, provenance, and response enforcement around that selector substrate. Exact upstream provenance and all five current Rust modifications are declared in native/vendor/NOTICE.md.

Registered evidence

On the separately registered Beta 2 AEB article_body tree, all 181 pages, the frozen Clusy path recorded F1 0.972127 versus 0.957546 for exact Python Trafilatura 2.1.0. The paired delta is +0.014581 with a 95% bootstrap interval of [+0.005547, +0.025336].

Because the candidate uses an rs-trafilatura descendant, this is a same-family implementation/version comparison, not evidence for an independently originated extraction algorithm or universal SOTA. The receipt also records 173.97 pages/s for machine-local in-memory extraction; that is not HTTP throughput, live crawl speed, or an SLA.

Run it

git clone https://github.com/clusy-io/clusy-crawler.git
cd clusy-crawler
git checkout v0.2.0-beta.4
cp .env.example .env
docker compose up --build --detach
curl --fail http://127.0.0.1:11235/health/ready

The default Compose service binds to loopback. Before exposing it, follow docs/SELF_HOSTING.md and set production authentication, fingerprinting, exact source identity, network policy, and browser sandbox prerequisites.

See CHANGELOG.md for the complete Beta 4 boundary and docs/BENCHMARKS.md for the registered receipt and limitations.

Clusy Crawler v0.2.0 Beta 3 Preview

Clusy Crawler v0.2.0 Beta 3 Preview

Fast, bounded, source-derived web extraction you can run on your own
infrastructure.

Beta 3 turns the optional quality path into a closed source-serialization
pipeline, hardens every runtime tier, and ships a release surface that is easy
to evaluate without a hosted account or model dependency.

Why try it

• Apache-2.0 and self-hosted; the deterministic path requires no model.
• Static, browser, and optional quality runtime profiles.
• Native Rust/PyO3 extraction with guarded HTTP/2 fetching and conditional
  Chromium rendering.
• Bounded URL, redirect, DOM, text, table, image, list, code, math, output,
  queue, and concurrency work.
• Source-derived quality selection with authenticated, replay-verified local
  serialization and deterministic fallback.
• Real container CI for authentication, SSRF rejection, browser sandboxing,
  configured quality extraction, dependency boundaries, and source identity.

Registered public result

On the separately registered Beta 2 AEB article_body tree (all 181 pages),
Clusy recorded F1 0.972127 versus 0.957546 for exact Trafilatura 2.1.0, a
paired delta of +0.014581 with a 95% bootstrap interval of
[+0.005547, +0.025336]. Machine-local in-memory extraction was
173.97 pages/s on the recorded machine.

That is a scoped, reproducible article-body comparison—not a universal web,
HTTP-service, or live-provider claim. The immutable receipt and retained
artifacts are in bench/evidence/.

Run it

git clone https://github.com/clusy-io/clusy-crawler.git
cd clusy-crawler
git checkout v0.2.0-beta.3
cp .env.example .env
docker compose up --build --detach
curl --fail http://127.0.0.1:11235/health/ready

Then send a URL to /crawl, /md, /html, or use /map for bounded
discovery. The default Compose service binds to loopback.

Release notes

See CHANGELOG.md
for the exact Added/Changed/Fixed boundaries and
docs/SELF_HOSTING.md
for production configuration.

Clusy Crawler v0.2.0 Beta 2 Preview

Clusy Crawler Beta 2 is a preview of the self-hosted FastAPI crawler and
extraction service.

What changed

• consistent 0.2.0b2 package, runtime, OpenAPI, and health identity;
• rewritten README, benchmark guide, and evidence status;
• one current registry-backed Article Extraction Benchmark receipt;
• fail-closed claim validation across Markdown, Unicode, nested headings, code
  fences, comparison language, deployment language, and reader-visible privacy
  paths;
• static, browser, and quality container boundaries verified in GitHub Actions.

Verified scoped evidence

A post-tag clean-source replay used a Git tree byte-identical to
v0.2.0-beta.2. On the fixed 181-page AEB article_body protocol:

• Clusy F1: 0.972127
• exact Trafilatura 2.1.0 F1: 0.957546
• F1 delta: +0.014581
• paired F1 delta CI95: [+0.005547, +0.025336]
• paired-bootstrap win fraction: 0.9996
• machine-local in-memory extraction throughput: 173.97 pages/s

The comparator ran in a separate 17-package hash-pinned, label-free
environment. The receipt is registered on main under
bench/evidence/aeb-article-body-trafilatura-2-1-77b8d00-beta2-public/.

This receipt covers local closed-loop article-body extraction only. Throughput
is one Darwin arm64 in-memory observation. It is not a universal SOTA,
live-vendor, HTTP-service-throughput, end-to-end crawling, production
deployment, stability, or reliability claim.

Validation

• Python: 1528 passed, 3 skipped
• Rust: 45 passed
• Ruff, mypy, documentation, evidence, Rust format/clippy, and all three
  container build/smoke boundaries passed in
  GitHub Actions run 30607087994

The existing Beta 1 tag remains available only as a superseded legacy evidence
record. See the current README and docs/BENCHMARKS.md on main for exact
hashes, limitations, and reproduction entry points.

Clusy Crawler v0.2.0 Beta 1 (Superseded)

Superseded: Use Clusy Crawler v0.2.0 Beta 2 Preview. Beta 1 is retained as a historical legacy evidence record; its source history is not the clean-root Beta 2 publication boundary.

Clusy Crawler 0.2.0 Beta 1 is the first public source preview of the
hardened extraction runtime.

This is a beta release. Pin this tag or an exact source commit; API and
operational compatibility may change before the stable release.

Included

• Native Rust/PyO3 extraction with a source-backed ordered document IR.
• Bounded static fetching, optional sandboxed Chromium rendering, and explicit
  provenance.
• Deterministic balanced, article_body, adaptive, and quality profiles
  with a fail-closed local fallback.
• Bounded recursive discovery, robots and same-site policy, SSRF defenses,
  authentication boundaries, and static/browser/quality container targets.
• Reproducible benchmark protocols, registered evidence, and checked-in claim
  boundaries.

Verified public result

On the registered, frozen 181-page AEB article_body protocol:

System	Precision	Recall	F1
Clusy	0.955147	0.989721	0.972127
Trafilatura 2.1.0	0.938372	0.977519	0.957546

• paired F1 delta: +0.014581
• paired-bootstrap 95% interval: [+0.005547, +0.025336]
• bootstrap win fraction: 0.9996 across 10,000 samples
• measured candidate throughput: 152.71 pages/s; p50 12.00 ms; p95
  24.05 ms; 0 errors
• registered claim:
  aeb.article-body.trafilatura-2-1.73b0297-public.2026-07-30

The comparator was isolated and hash-pinned. The protocol, report,
raw-artifact hashes, source inventory, native-binary identity, dependency lock,
and replay instructions are under bench/evidence/.

Validation

• GitHub Actions: lint/type/test and all container-build jobs passed.
• Python: 1364 passed, 3 skipped.
• Rust: 45 passed.
• Ruff, strict mypy, Rust format, documentation, dependency-lock, and
  benchmark-evidence validation passed.

Scope

The verified benchmark claim is limited to AEB article_body quality versus
exact Trafilatura 2.1.0 under the registered protocol. This release does not
claim universal SOTA, live Exa/Firecrawl superiority, service-level throughput,
or production deployment readiness. General-web, structured-data,
live-provider, hostile-input, and platform-canary evaluations continue.

No vendor output is used for training, distillation, or calibration.

See README.md, docs/SELF_HOSTING.md, and docs/BENCHMARKS.md for setup,
operations, evidence, and exact claim boundaries.