Skip to content

Clusy Crawler v0.2.0 Beta 4 Preview

Pre-release
Pre-release

Choose a tag to compare

View all tags
@ada-jl4025 ada-jl4025 released this 04 Aug 23:18
v0.2.0-beta.4
b220b43
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Clusy Crawler v0.2.0 Beta 4 Preview

Fast, bounded, source-derived web extraction that you can run on your own infrastructure and inspect end to end.

What ships

  • Static, browser, and optional quality images with explicit dependency boundaries.
  • Guarded HTTP/2 fetching, redirect-by-redirect SSRF checks, and conditional sandboxed Chromium.
  • Native Rust/PyO3 extraction, bounded local fallbacks, source-family specialists, deterministic discovery, and provenance-rich output.
  • Correct production token, Chromium sandbox, benchmark-replay, and archive-availability documentation.
  • The official Clusy wordmark, launch card, poster, and an 11.8-second X-ready animation.

Engine lineage

The generic HTML selector is Trafilatura-family software, not a from-scratch Clusy selector. The native path uses vendored rs-trafilatura broad 0.2.2 for the general candidate and a 9261e08-derived article backend for explicit article_body plus bounded adaptive rescue. The local fallback set includes Python Trafilatura 2.1.0, Readability, Markdownify, documentation-specific extraction, and raw-text rescue.

Clusy adds the guarded service, routing, specialists, candidate admission and comparison, structure recovery, Document IR and receipt primitives, crawl budgets, provenance, and response enforcement around that selector substrate. Exact upstream provenance and all five current Rust modifications are declared in native/vendor/NOTICE.md.

Registered evidence

On the separately registered Beta 2 AEB article_body tree, all 181 pages, the frozen Clusy path recorded F1 0.972127 versus 0.957546 for exact Python Trafilatura 2.1.0. The paired delta is +0.014581 with a 95% bootstrap interval of [+0.005547, +0.025336].

Because the candidate uses an rs-trafilatura descendant, this is a same-family implementation/version comparison, not evidence for an independently originated extraction algorithm or universal SOTA. The receipt also records 173.97 pages/s for machine-local in-memory extraction; that is not HTTP throughput, live crawl speed, or an SLA.

Run it

git clone https://github.com/clusy-io/clusy-crawler.git
cd clusy-crawler
git checkout v0.2.0-beta.4
cp .env.example .env
docker compose up --build --detach
curl --fail http://127.0.0.1:11235/health/ready

The default Compose service binds to loopback. Before exposing it, follow docs/SELF_HOSTING.md and set production authentication, fingerprinting, exact source identity, network policy, and browser sandbox prerequisites.

See CHANGELOG.md for the complete Beta 4 boundary and docs/BENCHMARKS.md for the registered receipt and limitations.

Assets 5
Loading