🌐 DonSeTch

Give your AI agent the web. Built from scratch in Rust. No keys, no accounts.

Fetch · search · crawl · bypass bot walls · read PDFs (even scanned) · semantic reranking One MCP server · Chrome's own TLS library · headless browser escalation · zero Python

⚠️ BETA — DO NOT USE IN PRODUCTION

DonSeTch is in active development. Key systems are being built, tested, and hardened. Things may break, APIs may change, features may be incomplete. Star the repo and check back. The release you want is coming.

cargo install donsetch

🚀 Install · 🎯 The 3 tools · 🛡️ Chrome TLS · 👻 Solve & Bounce · 🧠 Self-Improving · 🔎 Search · 🌐 Fetch · 🕷️ Crawl · 📄 PDF · 📊 Comparison · ⚠️ Gotchas · 🧱 Limits

---

DonSeTch is an MCP server that gives any AI agent (Claude Code, Cursor, OpenCode, anything that speaks MCP) full web research from a single local process.

Three tools. Zero API keys. Zero accounts. Built in Rust — one binary, no Python, no Playwright, no Selenium, no reqwest, no hyper. Every layer built from scratch. That's the point.

Speaks MCP 2024-11-05 through 2026-07-28. Works with every MCP client, old and new.

✨ What makes it different

Innovation	What it means	Status
🛡️ Real Chrome TLS	Drives Chrome's own BoringSSL natively — fingerprint is emergent from the real engine, not a faked table that rots. Your ClientHello IS Chrome's ClientHello.	Verified live
⏱️ Temporal stealth	Fakes the relationship over time with an origin — TLS session resumption, conditional revalidation (304), persistent cookies. The loudest remaining bot tell, and nobody else fakes it.	Verified live
👻 Solve-and-bounce	Browser solves the challenge, hands cookies to tier 1, goes to sleep. Tier 1 fetches at full speed. The browser almost never fetches content.	Live
🧠 Self-improving fetch	Learns from every fetch. Cookie lifetimes learned adaptively. Warm starts skip the browser entirely. Converges to optimal routing per domain.	Live
🔎 Semantic reranking	Local ONNX cross-encoder reads query + result through full attention. Pushes out generic Wikipedia articles that keyword-match but aren't about the topic.	A/B verified
🔑 Keyless search	10+ backends in parallel, fused by cross-engine consensus. No API keys, no accounts, no billing. $0 forever.	Live
📄 Pixel-fusion PDF	Glyphs + rendered pixels from the same stream, fused deterministically. No hallucination. Per-region trust audit. Scanned PDFs auto-OCR'd.	40-doc battle
🧬 Built from scratch	Own HTTP/2 (HPACK, flow control), own extraction engine, own PDF parser, own search aggregator, own crawl engine. Zero dependency on existing OSS web tooling.	249 tests

---

🛡️ Chrome TLS, not Chrome-like

Everyone in the impersonation game (curl-impersonate, rquest, wreq) patches a foreign TLS stack to resemble Chrome and ships hardcoded fingerprint tables that rot as browsers evolve.

DonSeTch does something different in kind:

We drive Chrome's own TLS library (BoringSSL) with its native Chrome behaviors switched on — GREASE, extension permutation, ECH-GREASE, ALPS, SCT, OCSP, brotli cert-compression. The ClientHello is generated by the same machinery that generates Chrome's.

When Chrome's TLS posture shifts, we update a data table, not patch a C library. The fingerprint isn't a table we fake. It's emergent from the real engine.

📊 Verified against live Chromium 150 at tls.peet.ws

Signal	Match
JA4	t13d1516h2_8daaf6152771_... — cipher hash identical to Chrome 150
Akamai h2 fingerprint	1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p — exact match
h2 header order	sec-ch-ua → sec-ch-ua-mobile → sec-ch-ua-platform → ... — exact match
Extension set	identical (contents differ only in random GREASE/key material, like real Chrome)

The _c suffix differs ONLY because Chrome 150 sends ML-DSA post-quantum sigalgs (0x904/5/6) and vendored BoringSSL 5.1.0 doesn't have them yet. Ours is genuine Chrome ≤131-family JA4.

🔧 Own HTTP/2 stack (because off-the-shelf leaks)

Off-the-shelf h2 (hyper's h2 crate) doesn't expose pseudo-header order, exact SETTINGS set, WINDOW_UPDATE values, or HPACK indexing strategy — all fingerprintable (Akamai h2 fingerprint).

So we wrote our own:

• 🧩 Own HPACK (RFC 7541 — tables generated from the RFC text, all 257 Huffman symbols + 61 static entries verified)
• 📦 Own frame engine (SETTINGS, HEADERS, DATA, WINDOW_UPDATE, PING, GOAWAY, RST_STREAM, CONTINUATION)
• 🔄 Own flow control with WINDOW_UPDATE replenishment
• 🔗 Own connection pool with TLS 1.3 session resumption

No reqwest. No hyper. No isahc. Every byte on the wire is ours.

⏱️ Temporal stealth (the tell nobody else fakes)

Everyone fakes the handshake. But a bot wall's second look is temporal: what does the client do over time?

Real Chrome resumes TLS sessions, holds h2 connections open, carries cookies across visits, and revalidates its cache (If-None-Match → 304 Not Modified).

Every scraper on earth does a full handshake, fresh connection, no cookies, no cache — every single time. That temporal silence is the loudest remaining tell.

DonSeTch fakes the relationship over time with an origin:

Mechanism	What it does	Why it matters
🔄 TLS session resumption	Per-origin session-ticket cache. Resumed handshakes are abbreviated.	Scrapers never resume. Chrome always does.
🔗 h2 connection pool	Connections kept alive, reused across fetches.	Fresh connection every time = bot signal.
✅ Conditional revalidation	Sends If-None-Match / If-Modified-Since on refetch. 304 → serve cached body.	Scrapers NEVER send conditional headers. Browsers always do. Nobody in this category does this.
🌐 Happy Eyeballs	Races IPv6 vs IPv4 with 250ms stagger.	Chrome does exactly this. Fixes dead-IP 10s timeouts.
🍪 Persistent cookie jar	Cookies survive across calls like a real browser profile.	A client with no cookie memory is a bot.

More stealth AND faster. The rare quadrant.

---

👻 Solve-and-bounce

The entire industry does tier 2 wrong: use a 600MB browser for everything. Every fetch — even ones plain HTTP could do — pays browser startup, browser RAM, browser CPU. And their stealth is subtractive: launch with automation flags, then inject JS patches to hide the damage. Every patch is a detectable lie.

DonSeTch inverts both.

The browser almost never fetches content. It exists to do exactly two things HTTP can't: pass JS challenges and execute JS-rendered pages. Its output is cookies (handed to tier 1, which fetches at full speed) or rendered HTML (handed to the extraction engine). Browser uptime is measured in seconds per domain per cookie-lifetime.

Step	What happens	Speed
1️⃣ Tier 1 fetch	Fast stealth HTTP (BoringSSL TLS)	~100-300ms
2️⃣ Wall detected	Cloudflare / DataDome / PerimeterX / Akamai	—
3️⃣ Ghost solves	Headless browser navigates, waits for challenge to clear, harvests clearance cookies	~2-6s
4️⃣ Bounce 🎾	Cookies handed to tier 1. Tier 1 re-fetches at full speed. Browser goes to sleep.	~100-300ms
5️⃣ Subsequent fetches	Tier 1 with warm cookies. Browser stays asleep.	~100-300ms

🧬 Nothing is patched because nothing is broken

Raw CDP launch without --enable-automation means navigator.webdriver is natively false. No JS injection ever. No Runtime, Console, or Debugger domains — the #1 modern CDP detection trap (castle.io). Environmental truthfulness instead of spoofing: real window, real GPU, real locale, consistent story.

💾 Process lifecycle — the RAM-smart part

The ghost process is SIGSTOP'd (frozen, not killed) after 20s of idleness:

State	RAM	CPU	Wake time
Active	full	real	—
Frozen (SIGSTOP)	mapped but cold (kernel swaps under pressure)	0.000	~50ms (SIGCONT)
Reaped (>10 min frozen)	freed	0	~1-2s (relaunch, profile keeps warmth)

Crash-transparent: thaw finds a dead browser → silent relaunch. The agent never sees the lifecycle.

Persistent profile dir ~/.cache/donsetch/ghost-profile: aged cookie/history state makes challenges EASIER (real users have history) and cf_clearance survives daemon restarts.

---

🧠 Self-improving fetch

Every fetch is both an action AND an observation. The system learns from each outcome and routes the next fetch more efficiently. The more you use DonSeTch, the less it escalates to tier 2.

No ML, no prediction models. Pure deterministic observation + state → better routing. The loop converges, it doesn't guess.

The loop

FETCH → OUTCOME → OBSERVE → UPDATE DOMAIN PROFILE → NEXT FETCH
  ↑                                                    ↓
  └────────────── ROUTE FROM PROFILE ──────────────────┘

Visit	Route decision	What happens	What the profile learns
🆕 Visit 1 (unknown)	Cold	Tier 1 → walled → ghost solves → cookies stored	needs_tier2=true, cookies vaulted
🔑 Visit 2 (cookies fresh)	Warm	Tier 1 with injected cookies → success	Cookies refreshed via write-back. Ghost stays asleep.
🔑 Visit 3 (cookies fresh)	Warm	Tier 1 with injected cookies → success	Session stays alive indefinitely as long as visits continue.
⏭️ Visit N (cookies expired)	SkipToSolve	Skip the doomed tier-1 round-trip, go straight to ghost	Saves one HTTP request + wall detection on every known-hard domain.
🔁 Visit M (24h since cold check)	RecheckCold	Try tier 1 cold — the wall may have been removed	If it succeeds: needs_tier2=false. Domain reverts to easy routing.

📈 Adaptive cookie lifetime learning

When warm cookies go stale (warm fetch gets walled), the system learns the real lifetime:

observed_lifetime = min(previous_observed, now - last_solved)

Next time, it re-solves at 18 min instead of wasting a fetch at 30 min that just hits the wall. Over multiple cycles, observed_lifetime converges to the real cookie lifetime for each domain. This is the self-improvement: the system gets better at predicting cookie death with each observation.

🔄 Cookie write-back (the session-keeper)

When tier 1 fetches with warm cookies and succeeds, the server may send fresh Set-Cookie headers. DonSeTch captures them and writes back to the on-disk profile.

Process restart → loads fresh cookies → warm tier 1 continues without a re-solve. Sessions stay alive as long as the agent keeps visiting, just like a real browser tab.

🛡️ Three-layer cookie freshness

A cookie is fresh only if ALL three layers agree:

Layer	Source	What it catches
1️⃣ Server-set expiry	expires_at from CDP or Max-Age	The cookie's claimed lifetime
2️⃣ Observed lifetime	Learned from warm-stale events	The cookie's REAL lifetime (often shorter than claimed)
3️⃣ TTL cap (2h hard limit)	Safety net	Server-side invalidation (IP change, session revoke)

---

🎯 The 3 tools

Tool	One-liner
🌐 fetch	Fetch any URL as clean markdown. HTTP first, escalates to headless browser if blocked. PDFs with OCR, focus for token savings, toc/section, pagination.
🔎 search	Keyless multi-engine web search. 10+ backends in parallel, consensus + semantic reranking. Returns URLs + snippets, not content.
🕷️ crawl	Best-first same-domain crawl. Sitemap + frontier, adaptive pacing, resume tokens. focus for budget management.

---

🔎 Keyless search

No API key, no account, no third-party service. 10+ keyless backends in parallel on your machine, merged, deduped, ranked.

• 🌐 10+ independent backends: Brave, Bing, DuckDuckGo, Mojeek + keyless verticals (GitHub, Wikipedia, HN, Semantic Scholar, arXiv, StackExchange, MDN, Google News). Six+ independent index families, not the same feed twice.
• 🧠 Semantic reranking: local ONNX cross-encoder (ms-marco-MiniLM-L-6-v2, 23MB, Apache-2.0) reads query + title + snippet through full transformer attention. It understands that "fast web scraper" and "high-speed crawler" are the same thing. BM25 can't. Blends 60/40 with RRF + BM25 + consensus. ~5ms/pair on CPU. Graceful no-op if model unavailable.
• 🎯 Cross-engine consensus: a URL returned by several independent indexes gets a consensus boost. Free authority signal. Every result carries score, consensus count, and engines list.
• 🛡️ Honest reporting: weak=true means low consensus — treat with care. Per-engine status (ok / blocked:429 / timeout / no-results) always visible. Never a fake "no results" that's actually a rate limit.

Why SearXNG loses and we don't

Problem	SearXNG	DonSeTch
Transport	python-requests — engines flag at TLS layer (Brave: 403/1015)	Own BoringSSL: Chrome-true ClientHello + h2. All engines answer 200.
Merge	Dedupes by URL, sums positions — bad engine's #1 beats good engine's #2	Weighted RRF + consensus + BM25 + domain priors + diversity cap
Rate limits	One IP	Egress pool (direct + proxies), per-engine health, query cache
Honesty	Doesn't report which engines answered	Per-engine status + weak flag — agents trust the output contract

🧪 A/B test: semantic reranking on niche queries

6 niche queries, A/B tested with reranker on vs off. 4 clear improvements, 1 correct no-change, 1 defensible change. Zero degradations.

Query	What the reranker did
"best lightweight linux distro for old laptops"	Demoted Stack Overflow's "for Development" result (tangential). Promoted on-topic results.
"how to prevent sql injection in php"	Pushed out a duplicate Stack Overflow page (same URL, HN metadata bolted on). Pulled in two new on-topic sources.
"neural network pruning techniques"	Pushed out Wikipedia: "Neural network" (generic, nothing about pruning). Pulled in a Springer academic paper.
"sustainable architecture materials"	Pushed out Wikipedia: "Sustainability" (generic). Promoted "Top 50 Sustainable Materials" over "15 Innovative Materials".
"what causes inflation in developing countries"	No change. RRF+BM25 already had optimal ordering. Cross-encoder agreed. Correct behavior.
"how to make sourdough bread at home"	Promoted wikiHow (how-to guide) over nickskitchen (recipe). The query asks "how to make" — a how-to, not a recipe.

Most consistent pattern: pushing out generic Wikipedia articles that keyword-match but are semantically tangential. RRF+BM25 can't catch this because "neural network" appears in both the query and the Wikipedia title. The cross-encoder reads the full title+snippet and recognizes that a generic neural network article isn't about pruning.

⚙️ Adaptive egress governor

Rate limits are a BUDGET problem, not a rotation problem. The governor never exceeds what each lane can sustain:

#	Mechanism	What it does
1	Per-engine trust EWMA	Each engine has a learned trust score (0.2..2.0). High-trust gets fan-out priority; low-trust cut first under stress.
2	Adaptive fan-out width	Healthy pool = all engines. Stressed (30%+) = max 3. Heavy (50%+) = max 2. Starved (65%+) = top engine + verticals.
3	Chronic-failure quarantine	3 consecutive failures = benched for 10 min.
4	Single-flight	Two identical in-flight queries share one leader's result. Agent parallel calls spend budget once.
5	Retry wave	Failed engines get one retry through a fresh egress — but only when the merge is thin.
6	Persistent disk cache	Queries cached with intent + recency-aware TTL. Survives restarts.
7	DONSEEK_PROXIES	Route engine requests through your proxies. Preflight benches dead lines.

---

🌐 Fetch

fetch tries plain HTTP first (~100-300ms). If the site serves a bot wall or a JS shell, it auto-escalates to the ghost browser, solves the challenge, bounces cookies back to tier 1, and re-fetches at full speed.

• 📝 DonSift extraction engine: HTML bytes in, agent-native markdown out. Not article text — a block model: typed blocks (Heading/Para/List/Table/Code/Quote/Media) with heading breadcrumbs. The block model powers BM25 focus, stable pagination, and token-war render policies.
• 🎯 focus — BM25-relevant blocks only. Cuts context 80%+ on long pages. 12-language BM25: CJK character unigrams + bigrams, 12-language stopword lists, light stemming, accent folding. Chinese "机器学习" → ['机','器','学','习','机器','器学','学习'].
• 📑 toc + section — heading outline first, then target one section. Two cheap calls instead of one expensive one. No other extractor offers this two-pass pattern.
• 📄 Pagination — next_offset in the response. Call again with offset=that value.
• ✂️ Token-war policies — links stripped by default (~30% savings), link-farm lists dropped, bare-link/bare-number lines dropped, wiki [edit] junk dropped, cross-block duplicate suppression.
• 🏷️ Content classification: Article / Listing / Forum / Docs / Table / Page from block composition.
• 📊 Quality score (0.0-1.0): content density, metadata, structure, language, text volume.
• 🔔 Agent-trust signals inline — focus-miss notice, section-miss notice, JS-shell warning, empty-content note — all in the content, not metadata.

🛡️ Anti-bot benchmark

Site	Protection	Status
Cloudflare-protected sites	Cloudflare interstitial	✅ 200 OK
DataDome sites	DataDome	✅ 200 OK
Stack Overflow	Cloudflare	✅ 200 OK
Medium	Cloudflare	✅ 200 OK
NowSecure	Cloudflare challenge	✅ 200 OK
Hacker News	None (baseline)	✅ 200 OK
Interactive captcha sites	hCaptcha / reCAPTCHA	⛔ Honest block

---

🕷️ Crawl

crawl walks same-domain links in best-first order. Two phases: sitemap discovery (cheap URL inventory in one fetch), then Governor-paced frontier walk with extraction per page.

• 🗺️ Three modes: full (default) = sitemap map + content. map = URL inventory only (very cheap). content = skip sitemap, BFS from seed.
• 🎯 Focus-ranked frontier: focus="query" ranks pages by BM25 relevance and crawls only matching ones. Essential for large sites.
• ⏱️ Adaptive pacing: the Governor paces per (host, lane). Success → steady. 429/503 → exponential. Error → cooldown.
• 🔄 Resume tokens: stopped crawls return a resume token. Call again with resume=token to continue. Valid 30 min, survives restarts.
• 📋 Near-dup detection: title + first 200 normalized chars → hash. Duplicates skipped, not re-extracted.
• 🛑 Honest stop reasons: FrontierEmpty ✅ (done), MaxPages/CharBudget/DepthLimit/Deadline ⏳ (use resume), ThrottledOut 🚫 (wait and resume).

---

📄 PDF + OCR

DonSeTch detects PDFs (by Content-Type or %PDF magic bytes) and parses them to structured markdown using a custom PDFium FFI. No external PDF library, no Python subprocess.

The insight nobody exploits

Every existing PDF extractor picks ONE modality and guesses about the other:

Approach	What it does	How it fails
📝 Heuristic (pymupdf4llm, pdfplumber)	Guesses layout from text-glyph positions/fonts	Collapses on anything visual: rule lines, borders, scans
🤖 ML pipeline (marker, docling, MinerU)	Throws layout-detection models at rendered pages	200MB-3GB models, and neural models hallucinate — emitted text is sampled, not guaranteed to exist

DonSeTch fuses both modalities, deterministically. PDFium gives us both a document's glyph stream (exact text + positions) AND rendered pixels (exact visual ground truth). Both come from the same content stream, so they're already aligned.

Pixels tell the truth about structure. Glyphs tell the truth about text. The fusion is exact because both render from one stream. No guessing, no hallucination.

Innovations

Innovation	What it does
🔲 Pixel-fusion rule extraction	Tables/borders/separators detected on the rendered bitmap via morphological opening. A rule line is a fact, not a hypothesis.
🔗 Span detection by ink continuity	A cell spans a separator iff the separator has NO ink under the cell's row band. Deterministic colspan/rowspan.
🔍 Trust audit + per-region arbitration	Glyph stream authoritative UNLESS: zero glyphs + pixels (scan), or region ≥30% PUA/garbage (broken ToUnicode). Corrupt regions get OCR'd from pixels even when neighbors read fine.
🔄 Orientation canonicalization	Vertical/rotated text = same document rotated. Rotate coordinate frames, run ONE pipeline. No special-case code.
✅ Confidence honesty	Verbatim glyphs or OCR with per-line confidence. [uncertain: ...] markers below threshold. Neural extractors cannot offer this.
📝 Forms as data	AcroForm widgets → name/type/value triples. Forms are only "hard" when treated as graphics.

Three-engine fusion (mirrors tier 1 → tier 2)

Tier	What	When
🟢 Tier A (always, zero extra bytes)	Glyph stream + pixel-fusion layout engine	Every page
🟡 Tier B (lazy: ONNX + PP-OCR models)	OCR for pages/regions with no trustworthy text	Scans, broken ToUnicode
🔴 Tier C (design hook)	Layout-detection models for low-confidence structure	Future

📊 PDF battle test results

40-document battle corpus, zero garbage output, 6-14x faster than Python alternatives. 120/120 fuzz clean.

Document type	Result
Academic papers	✅ Clean text — math symbols recovered, not CID garbage
Scanned documents	✅ OCR'd — PP-OCR cascade (En → Zh → Deva), confidence-scored
Tax forms (W-9)	✅ Forms as data — field names + values as table
Multi-column layouts	✅ Reading order preserved — column detection + merge
Encrypted PDFs	⛔ Honest flag — encrypted: password required
Corrupt PDFs	⛔ Honest flag — corrupt: parse failed at offset N
Nepali UDHR (broken ToUnicode)	✅ 10,542 usable Nepali chars at 86% confidence (pymupdf: 28 chars)

The Nepali UDHR is typeset in TimesNewRomanPSMT with broken ToUnicode (PUA remap). Font sniffing says "Latin." The fix is a confidence cascade: OCR page one with En → Zh → Deva recognizers, lock the winner by mean confidence for the rest. A document pymupdf reads as 28 characters comes out as 10,542.

---

🏗️ Built from scratch

Every layer built in Rust. No dependency on existing OSS web tooling.

Component	What it does	Key files
🛡️ DonShadow	Tier 1 stealth HTTP — BoringSSL TLS, own HTTP/1.1 + HTTP/2, temporal stealth, cookie jar, conditional revalidation	src/fetch/, src/transport/
👻 DonGhost	Tier 2 ghost browser — CDP (no Runtime/Console/Debugger), solve-and-bounce, SIGSTOP lifecycle, render mode	src/ghost/
📝 DonSift	HTML-to-markdown — block model, 12-language BM25 focus, token-war policies, content classification	src/extract/
🔎 DonSeek	Keyless multi-engine search — weighted RRF + BM25 + consensus + semantic reranking, adaptive egress governor	src/search/
🕷️ DonTread	Crawl engine — sitemap, focus-ranked frontier, Governor pacing, resume tokens, near-dup detection	src/crawl/
📄 DonSheet	PDF extraction — PDFium FFI, pixel-truth fusion, OCR arbitration cascade, forms as data	src/pdf/
🔌 MCP daemon	stdio server — JSON-RPC 2.0, MCP 2024-11-05 through 2026-07-28, 3 tools at ~1.8K tokens	src/mcp/

249 tests. Zero clippy warnings. cargo clippy --release -- -Dwarnings is the law.

---

🚀 Install

Prerequisites

Dependency	Why	Linux	macOS	Windows
Rust 1.75+	Build toolchain	rustup	rustup	rustup
Go 1.22+	BoringSSL build	pacman -S go	brew install go	winget install GoLang.Go
NASM	BoringSSL assembly	pacman -S nasm	brew install nasm	choco install nasm
LLVM/Clang	bindgen headers	pre-installed	pre-installed	choco install llvm
CMake	BoringSSL build	pacman -S cmake	brew install cmake	winget install cmake
Chromium (optional)	Tier 2 browser	pacman -S chromium	brew install chromium	Edge works

Build

git clone https://github.com/dondai44423/donsetch.git
cd donsetch
cargo build --release

Binary lands at target/release/donsetch. First build takes ~5 min (compiling BoringSSL). Subsequent builds are cached.

📋 Build notes

• BoringSSL is vendored and built from source via boring-sys. First build compiles it (~5 min), then it's cached.
• PDFium is downloaded as a static library by build.rs — no manual setup.
• ONNX Runtime is downloaded by oar-ocr (OCR) and ort (reranker) at build time.
• Models (OCR + reranker) download on first use to ~/.cache/donsetch/, not bundled in the binary.
• Feature flags: default = ["ocr", "rerank"]. Build with --no-default-features for HTTP-only (no OCR, no reranker, smaller binary).
• Cross-compilation: rustup target add <target>. Platform code is #[cfg]-gated, not forked.

Connect your AI agent

MCP over stdio — point your agent at the binary:

{ "mcpServers": { "donsetch": { "command": "/path/to/donsetch" } } }

No arguments, no API keys, no environment variables. Done.

⚙️ Optional environment variables

Env Var	What it does
DONSEEK_PROXIES	Comma-separated proxies for search engines. Preflight benches dead lines.
DONGHOST_DEBUG	Print ghost solve/render debug to stderr.
DONSHEET_DEBUG	Print PDF/OCR debug to stderr.
DONSHEET_DEBUG_CHARS	Print PDF layout character stream.

---

📊 Comparison

	DonSeTch	Hound	Crawl4AI	Jina Reader	Firecrawl
Language	Rust	Python	Python	Python (API)	TypeScript
TLS fingerprint	Real Chrome (BoringSSL, driven natively)	curl-impersonate (patched)	requests	their servers	their servers
HTTP/2 stack	Own (HPACK, flow control, session resumption)	primp	requests	their servers	their servers
Temporal stealth	✅ (session resumption, revalidation, cookies)	❌	❌	❌	❌
Tier 2 strategy	Solve-and-bounce (cookies to tier 1, browser sleeps)	Browser fetches everything	n/a	n/a	n/a
Self-improving	✅ (adaptive cookie lifetime, warm start, write-back)	❌	❌	❌	❌
Web search	✅ (keyless, 10+)	✅ (keyless, 10)	❌	✅	❌
Semantic reranking	✅ (local ONNX cross-encoder)	✅ (local ONNX)	❌	❌	❌
Deep crawl	✅ (resume tokens)	✅	✅	❌	✅ (cloud)
PDF → markdown	✅ (pixel-fusion, per-region arbitration)	✅ (pdfplumber)	partial	✅ (native)	✅ (cloud)
Scanned-PDF OCR	✅ (PP-OCR, confidence cascade)	✅ (rapidocr)	❌	❌	✅ (paid)
Query focus	✅ (12-language BM25, CJK bigrams)	✅ (BM25)	✅ (BM25)	❌	❌
Runs locally	✅	✅	✅	❌	self-host
MCP server	✅	✅	community	✅	build it
Token cost	~1.8K (3 tools)	~2.7K (6 tools)	varies	n/a	varies
License	AGPL v3	MIT	Apache 2.0	proprietary	MIT

---

⚠️ Gotchas

Surprise	Why
🔨 First build takes ~5 min	BoringSSL is compiled from source. Cached after that.
🐹 Go is a build dependency	BoringSSL's build system is Go-based. You need Go even though DonSeTch is Rust.
⛔ Interactive captchas not solved	hCaptcha, reCAPTCHA, Turnstile checkbox = honest dead end. No solving service by design.
🤖 robots.txt ON by default for crawl	respect_robots=true for crawl. fetch doesn't check robots.
🌐 Search rate-limits without a proxy	Keyless search scrapes public engines from your IP. Set DONSEEK_PROXIES for heavy use.
📦 Not built for mass scraping	DonSeTch is for agentic research, not bulk extraction.

---

🧱 Honest limits

What it can NOT do	Why
⛔ Solve CAPTCHAs	Deliberate. You get a clear error, not a hang.
🔐 Sites requiring login	Out of scope (page rendering, not authenticated sessions).
🔬 ML-DSA post-quantum signatures	BoringSSL 5.1.0 lacks them. Will be added when BoringSSL gains it.
🪟 Windows/macOS PDF CI	Compiled but CI verification pending. Linux is primary target.
📡 Search with all engines down	Returns an error with per-engine status. Honest, not fake.

---

🤝 Contributing

PRs welcome. See CONTRIBUTING.md. Run cargo clippy --release -- -Dwarnings and cargo test --release before submitting. AGPL v3 — all contributions under the same license.

📄 License

Copyright (c) 2026 Bishesh Bhandari. AGPL-3.0 — see LICENSE.

---

If DonSeTch saves you time, ⭐ the repo

AGPL v3 · Changelog · Issues · Releases