Skip to content

Clusy Crawler v0.2.0 Beta 1 (Superseded)

Pre-release
Pre-release

Choose a tag to compare

View all tags
@ada-jl4025 ada-jl4025 released this 30 Jul 23:37
· 16 commits to main since this release
v0.2.0-beta.1
c39a6f9

Superseded: Use Clusy Crawler v0.2.0 Beta 2 Preview. Beta 1 is retained as a historical legacy evidence record; its source history is not the clean-root Beta 2 publication boundary.

Clusy Crawler 0.2.0 Beta 1 is the first public source preview of the
hardened extraction runtime.

This is a beta release. Pin this tag or an exact source commit; API and
operational compatibility may change before the stable release.

Included

  • Native Rust/PyO3 extraction with a source-backed ordered document IR.
  • Bounded static fetching, optional sandboxed Chromium rendering, and explicit
    provenance.
  • Deterministic balanced, article_body, adaptive, and quality profiles
    with a fail-closed local fallback.
  • Bounded recursive discovery, robots and same-site policy, SSRF defenses,
    authentication boundaries, and static/browser/quality container targets.
  • Reproducible benchmark protocols, registered evidence, and checked-in claim
    boundaries.

Verified public result

On the registered, frozen 181-page AEB article_body protocol:

System Precision Recall F1
Clusy 0.955147 0.989721 0.972127
Trafilatura 2.1.0 0.938372 0.977519 0.957546
  • paired F1 delta: +0.014581
  • paired-bootstrap 95% interval: [+0.005547, +0.025336]
  • bootstrap win fraction: 0.9996 across 10,000 samples
  • measured candidate throughput: 152.71 pages/s; p50 12.00 ms; p95
    24.05 ms; 0 errors
  • registered claim:
    aeb.article-body.trafilatura-2-1.73b0297-public.2026-07-30

The comparator was isolated and hash-pinned. The protocol, report,
raw-artifact hashes, source inventory, native-binary identity, dependency lock,
and replay instructions are under bench/evidence/.

Validation

  • GitHub Actions: lint/type/test and all container-build jobs passed.
  • Python: 1364 passed, 3 skipped.
  • Rust: 45 passed.
  • Ruff, strict mypy, Rust format, documentation, dependency-lock, and
    benchmark-evidence validation passed.

Scope

The verified benchmark claim is limited to AEB article_body quality versus
exact Trafilatura 2.1.0 under the registered protocol. This release does not
claim universal SOTA, live Exa/Firecrawl superiority, service-level throughput,
or production deployment readiness. General-web, structured-data,
live-provider, hostile-input, and platform-canary evaluations continue.

No vendor output is used for training, distillation, or calibration.

See README.md, docs/SELF_HOSTING.md, and docs/BENCHMARKS.md for setup,
operations, evidence, and exact claim boundaries.

Assets 2
Loading