Release v2.52.2 — CLI authenticate for Spider Cloud · spider-rs/spider

Changes since vv1.7.5

fix: concurrency/memory/panic audit hardening (v2.52.7) (#401)
chore(release): bump workspace to v2.52.6
browser: per-crawl control map for optional chrome enhancements (#400)
chore(release): bump workspace to v2.52.5
browser: genericize internal gateway topology/metrics in doc comments
browser: harden in-request gateway re-acquire (jitter, opt-out, timeouts)
security: fix cargo-audit failures (quick-xml DoS + lopdf)
chore(deps): bump actions/checkout from 4 to 7 (#399)
security: bump quinn-proto to 0.11.15 (RUSTSEC-2026-0185)
browser: connection reliability and failover improvements
chore(release): v2.52.4 — UTM-tag spider.cloud README links
chore(release): v2.52.3 — markdown scrape default, api cloud mode, branded login page
feat(cli): markdown scrape by default + branded sign-in page
chore(release): bump workspace to v2.52.2
feat(cli): browser OAuth login + MCP OAuth client example
chore(release): bump workspace to v2.52.1
fix(website): bound sitemap crawl task spawning with semaphore (#397)
feat(chrome_intercept): with_remote_local_policy (spider 2.52.0) (#396)
Fix 500ms timeout for proxy error statuses 598/599 (#395)
chore(release): bump workspace to v2.51.208
fix(website): crawl_raw uses raw sitemap chain on chrome builds (#393)
feat: add on_link_blocked_callback for robots.txt blocked URLs (#394)
perf(page,website): drop redundant heap allocs on link-validation paths (v2.51.207)
fix(chrome): close leaked tabs in smart-upgrade + spawn-pages paths (v2.51.206)
fix(agent,mcp): bound page tabs, MCP sessions, and agent KV stores (v2.51.205)
docs: tighten README and rework intro
feat(agent,scripting): client reuse + usage tracking (v2.51.204)
perf(agent,scripting): per-worker interpreter cache + permit timeout (v2.51.203)
chore(release): v2.51.202 — patch bump
chore(release): v2.51.201 — formalize AdaptiveSemaphore safety contract
feat(website): AdaptiveSemaphore + with_adaptive_concurrency hook for runtime concurrency
feat(agent): pure-Rust embedded Python + JS scripting (v2.51.200)
feat(website): RemoteFetcher hook for delegated per-URL fetch (v2.51.198)
chore(release): sync internal cross-crate deps to v2.51.197
chore(release): v2.51.197 — chrome wait_for cap removal in non-success path
fix(chrome): honour caller request_timeout in non-run_events wait branch
chore(release): v2.51.196 — per-website decentralized worker URL overrides
chore(release): sync internal cross-crate deps to v2.51.195
chore(release): v2.51.195 — wreq network_interface plumbing
feat(wreq): wire network_interface through wreq configure_base_client
chore(release): sync internal cross-crate deps to v2.51.194
chore(release): v2.51.194 — wreq emulation header survival fix
fix(wreq): skip default_headers(empty) so emulation headers survive
chore(release): sync internal cross-crate deps to v2.51.193
chore(release): v2.51.193 — chrome_store_page tab keep-alive
fix(website): keep chrome tab alive for chrome_store_page subscribers
chore(examples): track chrome_proxy_dns_shortcircuit source
chore(release): v2.51.192 — chrome hedge stack-overflow fix
fix(page): heap-pin chrome DNS hedge future to stop crawl stack overflow
chore(release): update Cargo.lock for v2.51.191
chore(release): v2.51.191 — SSRF redirect guard + fmt
fix(redirect): block SSRF redirect targets in all follow policies
feat(page): expand IGNORE_EXTENSIONS + spool-safe chrome tunnel DNS confirm — v2.51.190
perf(page): cross-feature NXDOMAIN cache + DNS-probe & CDP-subscribe skip — v2.51.189
chore(examples): unregister chrome_proxy_dns_shortcircuit (untracked source)
test(example): chrome_nxdomain_test with CHROME_WS_URL plumbing
chore(release): sync internal cross-crate deps to v2.51.188
feat(chrome): CDP loadingFailed hedge trigger + v2.51.188
chore(release): sync internal cross-crate deps to v2.51.187
feat(chrome): DNS hedge alongside chrome navigation + v2.51.187
chore(release): sync internal cross-crate deps to v2.51.186
chore(release): cargo fmt + sync sister crates to v2.51.186
feat(chrome): two-signal NXDOMAIN cache + v2.51.186 (supersedes v2.51.185 pre-flight)
feat(chrome): opt-in pre-flight local DNS shortcircuit + v2.51.185
feat(solvers): detect embedded Cloudflare Turnstile widgets + v2.51.184
docs(readme): rewrite for clarity, lead with streaming + benchmarks
chore(release): sync internal cross-crate deps to v2.51.183
release(spider): v2.51.183 — chain-walk classifier for cache-wrapped errors
chore(release): sync internal cross-crate deps to v2.51.182
release(spider): v2.51.182 + regression test for cache-gate fix
fix(client): gate http-cache-reqwest middleware on configuration.cache
chore(release): sync internal cross-crate deps to v2.51.181
perf(page,utils): swap tokio yield_now for consume_budget in CPU-bound chunk loops + v2.51.181
chore(release): sync internal cross-crate deps to v2.51.180
fix(website): smart-mode chrome fallback for transport-level HTTP failures + v2.51.180
chore(release): sync internal cross-crate deps to v2.51.179
chore: silence pre-existing clippy dead-code warnings + v2.51.179
chore(release): bump remaining workspace crates to v2.51.178
chore(release): sync internal cross-crate deps to v2.51.178
fix(page): wire two-signal NXDOMAIN upgrade into Page::new_page_streaming + v2.51.178
fix(page): don't cache TimedOut + bump confirm_tunnel timeouts to 1500ms + v2.51.176
feat(page): chrome two-signal NXDOMAIN upgrade for empty-200 → 504 reclass + v2.51.175
feat(utils): wire two-signal NXDOMAIN upgrade into all fetch_page_html error paths + v2.51.174
chore(release): sync internal cross-crate deps to v2.51.173
feat(page): SOCKS proxy CONNECT failure detection + v2.51.173
chore(page): fix clippy::doc_lazy_continuation in two-signal docstring
chore(examples): add dns_nxdomain_test + chrome_remote_logrocket repros
test(page): gate non-cache helper tests off cache_request feature
chore(release): sync internal cross-crate deps to v2.51.172
feat(page): two-signal upgrade for cache_request 526 + v2.51.172
chore(release): sync internal cross-crate deps to v2.51.171
perf(page): swap HostDnsCache RwLock for DashMap (no user-facing locks) + v2.51.171
chore(release): sync internal cross-crate deps to v2.51.170
perf(page): bounded process-wide host DNS cache for tunnel-confirmation lookups + v2.51.170
chore(release): sync internal cross-crate deps to v2.51.169
feat(page): chrome path two-signal proxy tunnel classification + v2.51.169
chore(release): sync internal cross-crate deps to v2.51.168
feat(page): two-signal proxy tunnel classification via local DNS confirmation + v2.51.168
chore(release): sync internal cross-crate deps to v2.51.167
revert(page): tunnel/proxy CONNECT failures stay retryable + v2.51.167
chore(release): sync internal cross-crate deps to v2.51.166
fix(page): chrome ERR_TUNNEL_CONNECTION_FAILED / ERR_PROXY_CONNECTION_FAILED → 526 + v2.51.166
chore(release): sync internal cross-crate deps to v2.51.165
fix(page): classify proxy CONNECT tunnel failures as 526 (non-retryable) + v2.51.165
chore(release): sync internal cross-crate deps to v2.51.164
chore(release): bump workspace to v2.51.164
fix(config): http first-byte watchdog fires for single-proxy callers
chore(release): sync internal cross-crate deps to v2.51.163
fix(parallel_backends): short-circuit grace-period on permanent target failures + v2.51.163
chore(release): sync internal cross-crate deps to v2.51.162
fix(page): walk source chain in is_dns_error to catch macOS GAI NXDOMAIN + v2.51.162
chore(release): sync internal cross-crate deps to v2.51.161
fix(chrome): preserve HTTP final_redirect_destination across smart upgrade + v2.51.161
chore(release): sync internal cross-crate deps to v2.51.160
fix(website): apply redirect-aware selectors before smart establish link extraction + v2.51.160
chore(release): sync internal cross-crate deps to v2.51.159
fix(page): classify cache-wrapped transport errors as 526 + reorder DNS check + v2.51.159
chore(release): sync internal cross-crate deps to v2.51.158
fix(website): gate first-party allow builders on chrome feature + v2.51.158
feat(intercept): expose chromey 2.49 allow_first_party_* flags via Website builder
style: clippy fixes (manual_pattern_char_comparison, question_mark)
chore(ci): bump softprops/action-gh-release to v3
chore(release): sync internal cross-crate deps to v2.51.157
chore(release): bump workspace to v2.51.157
style: cargo fmt on new proxy_strategy + utils + website hunks
feat(proxy_strategy): trait + Website hook + lazy secondary client
feat(configuration): ProxyKind enum + proxies_by_kind sidecar
feat(utils): media_asset classifier + LazyArc primitive
chore(utils): solver-gate diagnostic logging in fetch_page_html_chrome_base
chore(release): sync internal cross-crate deps to v2.51.156
fix(chrome,validation): smart-mouse jitter loop + PerimeterX title parity
feat(chrome): route el.click() through smart-mouse path — v2.51.156
fix(chrome): route point-based clicks/drags through smart-mouse paths
chore(release): sync internal cross-crate deps to v2.51.155
fix(utils): handle_base must not byte-slice on UTF-8 boundaries — v2.51.155
ci(bench): install spider_worker before decentralized bench
fix(ci): bump hickory 0.26.0->0.26.1, ignore unfixable 0.25.x advisories, fix worker
chore(release): sync internal cross-crate deps to v2.51.154
fix(chrome): re-honor quiet_ms in wait-for-DOM v2 to handle bursty SPAs + v2.51.154
chore(release): sync internal cross-crate deps to v2.51.153
fix(utils): never cache failed responses + v2.51.153
chore(release): sync internal cross-crate deps to v2.51.152
fix(http): balance vitals counter on first-byte timeout early-return + v2.51.152
chore(release): sync internal cross-crate deps to v2.51.151
feat(http): wire first-byte watchdog into Page::new_page_streaming + crawl_smart auto-arm + v2.51.151
style: cargo fmt + targeted clippy fixes (drop redundant &'static, single-line log)
chore(release): sync internal cross-crate deps to v2.51.150
feat(http): auto-arm first-byte watchdog under balance + 2 HTTP proxies + v2.51.150
chore(release): sync internal cross-crate deps to v2.51.149
feat(http): first-byte watchdog foundation + v2.51.149
chore(release): sync internal cross-crate deps to v2.51.148
chore(release): bump workspace to v2.51.148
fix(chrome,hedge): track hedge's own connected_url + browser_dead
chore(release): sync internal cross-crate deps to v2.51.147
chore(release): bump workspace to v2.51.147
test(chrome): cooldown deadline math + last-write-wins on mark_url_bad
chore(release): sync internal cross-crate deps to v2.51.146
feat(chrome): first-byte watchdog jitter + measurement test + v2.51.146
chore(release): sync internal cross-crate deps to v2.51.145
feat(chrome): mark-bad endpoint rotation + auto-populated chrome_endpoint_url + v2.51.145
feat(chrome): first-byte watchdog + browser_dead signal for backend rotation
perf(chrome): lazy-cache multi-endpoint failover across setup_browser_configuration
chore(release): sync internal cross-crate deps to v2.51.144
fix(detect): trim leading whitespace before binary/asset sniffing + v2.51.144
chore(release): sync internal cross-crate deps to v2.51.143
chore(release): lockfile dep refresh + v2.51.143
chore(release): commit + push internal dep sync before publishing
chore(release): sync internal cross-crate deps to v2.51.142
chore(release): cargo fmt + lockfile refresh + v2.51.142
chore(release): sync internal cross-crate deps to v2.51.141
fix(chrome): honor request_timeout as a hard wall-clock cap + v2.51.141
feat(chrome_remote_cache): main-doc-only dump knob + v2.51.140
chore(release): sync internal cross-crate deps to v2.51.139
fix(utils): reclassify PerimeterX block-page signals + v2.51.139
chore(release): v2.51.138 — sync workspace versions for wait_for_idle_dom signal-stack
perf(chrome): signal-stack wait_for_idle_dom — RAF-cadence, no quiet timer
perf(page): metadata-first spool short-circuit + helpers + v2.51.137
fix(page): reclassify empty 200 responses to 504 + v2.51.136
chore(release): sync cross-crate deps to v2.51.135
fix(chrome): strip JSON quotes from CDP-captured response headers + v2.51.135
chore(release): sync cross-crate deps to v2.51.134
perf(chrome): skip streaming extraction on assets + v2.51.134
chore(release): sync cross-crate deps to v2.51.133
fix(smart): preserve legacy link-set semantics on chrome upgrade + v2.51.133
chore(release): sync cross-crate deps to v2.51.132
perf(smart): stream link extraction during chrome upgrade + v2.51.132
chore(release): sync cross-crate deps to v2.51.131
perf(chrome): fold link extraction into chunk pump (single-pass) + v2.51.131
perf(chrome): skip redundant fill-fetch when HTML extraction already ran + v2.51.130
perf(utils,page): pre-compute is_valid_utf8 during streaming + v2.51.129
fix(retry): drop h2 INTERNAL_ERROR from retryable reasons + v2.51.128
perf(chrome): switch SSL/cert detector trio to single-pass Aho-Corasick + v2.51.127
fix(retry): widen permanent-failure classification for HTTP/2 + 5xx + Chrome URL family + v2.51.126
fix(retry): classify SSL handshake failures as 526 across HTTP + Chrome + v2.51.125
chore(release): sync cross-crate deps to v2.51.124
chore(release): bump workspace to v2.51.124
chore(release): sync cross-crate deps to v2.51.123
fix(chrome): single-URL with_chrome_connections takes default path + v2.51.123
chore(release): sync cross-crate deps to v2.51.122
chore(release): bump workspace to v2.51.122
feat(page): classify address-unreachable + connection-refused as permanent
perf(chrome): batch Fetch.disable cleanup + LLM JS wrapper robustness + v2.51.121
feat(remote_cache): runtime config for skip_browser HTTP dumps
chore(release): bump workspace to v2.51.119
perf(parallel_backends): skip HTML-shape probes on assets in html_quality_score
test(smart): live smoke test covering v2.51.117 source_str→target_url refactor
chore(release): bump workspace to v2.51.117
perf(utils): drop redundant HTML UTF-8 decode in fetch_page_html_chrome_base
fix(website): emit synthetic Page to subscribers on chrome tab-creation timeout; bump to v2.51.116
chore(release): bump workspace to v2.51.115
fix(chrome_remote_cache): bypass proxies on remote cache HTTP client
feat(chrome_remote_cache): read-only mode + v2.51.114
chore(release): bump workspace to v2.51.113
chore(release): bump workspace to v2.51.112
perf(uring_fs): round-robin worker pool for parallel ring submission + v2.51.111
chore(release): bump workspace to v2.51.110
chore(release): bump workspace to v2.51.109
chore(release): bump workspace to v2.51.108
fix(website): use page.size() for initial_html_length on balanced spools + v2.51.107
fix(website): retry links on semaphore close + spool-aware JS challenge detect + v2.51.106
perf(chrome): lock-free OnceLock get/set init for tab-closer + v2.51.105
feat(chrome,hedge,parallel_backends): concurrent tab-closer + explicit context dispose + v2.51.104
fix(parallel_backends,chrome): close CDP tabs + background tab-closer watcher + v2.51.103
chore(tests): move logrocket network probe out of tree
chore(release): sync internal deps to v2.51.102
feat(tests): live network map + markdown integrity check for logrocket + v2.51.102
chore(release): sync internal deps to v2.51.101
feat(website): expose with_max_main_frame_navigations on Website + v2.51.101
chore(release): sync internal deps to v2.51.100
feat(chrome): navigation-loop guard for JS / meta-refresh loops + v2.51.100
chore(release): sync internal deps to v2.51.99
feat(chrome): enforce redirect_limit on chromey path (opt-in) + v2.51.99
chore(release): v2.51.98 — bump spider_network_blocker 0.0.81 → 0.0.82
chore(release): v2.51.97 — bump async-openai 0.33→0.35, tiktoken-rs 0.9→0.11
chore(release): v2.51.96
chore(release): v2.51.95
fix(page): webdriver+chrome+basic build under Page Drop impl
chore(release): v2.51.94
chore(release): sync internal deps to v2.51.93
chore(release): v2.51.93
fix: classify permanent DNS failures as 525 across http + chrome
chore(release): v2.51.92
chore(templates): add apache forbidden updated template
fix(website): sitemaps push now compiles under default features
perf,fix: eliminate wasteful clones and fix 5 test failures
perf(website): consume CaseInsensitiveString via into_string()
perf(utils,website,chrome): drop redundant to_string() and clones
chore: bump workspace to 2.51.86 after cargo update
perf(page,utils): eliminate wasteful clones in metadata and cache paths
perf(page): borrow external_domains_caseless in link-stream rewriters
chore: sync spider_agent internal deps to 2.51.83
fix(chrome): update remaining Page::new call sites missed in 2.51.82
chore: sync internal spider_agent deps to 2.51.82
perf(chrome): bundle Chrome fetch params into borrowed struct for inlining
fix(balance,chrome): cap failure-fallback wall time to request_timeout
fix(crawl): missing .await on channel_send_page lost pages to subscribers
feat(balance): per-Website lazy spool tmpdir with drop cleanup
feat(balance,chrome): wire direct-to-disk spool into fetch_page_html_chrome_base
perf(balance): DoS-cap the streaming spool writer + audit with_capacity sites
feat(balance): signature-equivalent spool infrastructure; defer fetch wire
feat(balance,chrome): stream-to-disk infrastructure for chrome spooling
feat(chrome,balance): adaptive chrome HTML streaming via chromey 2.46.40
perf(balance): streaming spool writer with inline vitals + full disk streaming
chore: bump all crates to v2.51.72
fix(page): drop unwrap in balance bytes tracking, bump crates to v2.51.71
chore: bump sibling crates to v2.51.70 to match spider release
feat(retry): add per-attempt hedge override on RetryDirective, fix E0716 in retry loop, bump v2.51.70
chore: sync internal crate dependency versions to 2.51.69
perf: pre-allocate Vec/HashMap/HashSet with known capacity, bump all crates to v2.51.69
feat(retry): add configurable RetryStrategy trait for dynamic per-attempt profile switching, bump all crates to v2.51.68
feat(page): add Page::size() for zero-cost content byte length, bump all crates to v2.51.67
chore: sync internal crate dependency versions to 2.51.66
fix(page): refuse to spool Chrome empty-HTML shells, bump all crates to v2.51.66
chore: sync internal crate dependency versions to 2.51.65
fix(warc): emit HTTP response headers so WARC files replay in replayweb.page (#384), bump all crates to v2.51.65
feat(chrome): restore Chrome hedge path (HedgeBrowser + race logic), bump all crates to v2.51.64
chore: bump all crates to v2.51.63
feat(chrome): add disable_javascript config to suppress JS execution via CDP, bump all crates to v2.51.62
chore: bump all crates to v2.51.61
chore: cargo fmt, cargo update, bump all crates to v2.51.60
test(chrome): add concurrency e2e tests for Chrome crawling
fix(chrome): send shutdown signal immediately after page close, fix e2e test URL
fix(chrome): prevent deadlock in goto_with_html_once when chrome_intercept enabled
chore: bump all crates to v2.51.59
chore(audit): suppress rustls-webpki and rand advisories
chore: bump all crates to v2.51.58, spider_remote_cache to v0.3
fix(cache): eagerly init worker, remove redundant set_client + v2.51.56
fix(cache): fix chrome_remote_cache compilation + v2.51.56
fix(cache): fix chrome_remote_cache compilation — cfg-gate tracing, bump chromey
perf(cache): use spider_remote_cache for batched concurrent uploads + v2.51.55
feat(chrome): WaitForLoad/WaitForNetworkIdle/WaitForNetworkAlmostIdle use CDP events + v2.51.54
feat(chrome): add WaitForLoad, WaitForNetworkIdle, WaitForNetworkAlmostIdle actions + v2.51.53
bump all crates to v2.51.52
perf(chrome): concurrent page_wait — two-phase network-then-DOM + v2.51.51
fix(balance): eliminate byte-counter leak causing Chrome crawl deadlock + v2.51.50
feat(spider): add wait_guard feature — adaptive wait-for timeout for chronic bad domains + v2.51.49
chromey v2.46.21, bump all crates to v2.51.48
bump all crates to v2.51.47
bump all crates to v2.51.46
fix(chrome): add backoff to single-endpoint retry loop, cap unbounded network waits + v2.51.45
bump all crates to v2.51.44
perf(content): SIMD-accelerate estimate_text_length, estimate_tag_bytes, estimate_base64_bytes + v2.51.43
chromey v2.46.13 -> v2.46.15, bump all crates to v2.51.42
bump all crates to v2.51.41
chromey v2.46.12 -> v2.46.13
fix(hedge+chrome): remove chrome hedge path, fix backend semaphore deadlock, skip warmup hedges + v2.51.40
perf(clean_html): zero-cost catch_unwind via AssertUnwindSafe, drop html clone + v2.51.39
fix(stability): catch lol_html rewrite_str UTF-8 panic in clean_html_* + v2.51.38
fix(balance): close byte-tracking memory leak in spool budget counter + v2.51.37
perf(balance): zero-disk-IO link extraction for spooled pages, spool-aware binary detection + v2.51.36
perf(simd): trie-dispatch content type classifier, phf hop-by-hop header filter + v2.51.35
perf(simd): zero-alloc asset extension check, memchr robotparser, branch-free networking_capable + v2.51.34
fix(concurrency): eliminate blocking paths — Notify-based ChannelGuard, TabCloseGuard timeout, eager abort_all + v2.51.33
chore: auto-sync internal dep versions in release.sh
chore: fix all internal dep versions, bump to v2.51.32
chore: fix spider_agent dep versions, bump to v2.51.31
fix(chrome): close leaked tabs preventing hedge deadlocks, remove forced retry + v2.51.30
chore: regenerate Cargo.lock for v2.51.29
refactor(parallel_backends): remove LightPanda references, rename to Cdp + v2.51.29
chore: regenerate Cargo.lock for v2.51.28
fix(parallel_backends): eliminate redundant grace period + double stat recording + v2.51.28
chore: regenerate Cargo.lock for v2.51.27
fix(parallel_backends): restore RNG primary/backend launch swap (0-1ms) + v2.51.27
chore: regenerate Cargo.lock for v2.51.26
perf(parallel_backends): true primary-vs-backend racing + remove primary jitter + v2.51.26
chore: regenerate Cargo.lock for v2.51.25
perf(uring_fs): unified read_file_chunked with auto uring/tokio fallback + v2.51.25
chore: regenerate Cargo.lock for v2.51.24
fix(balance): uring_fs fallback to tokio::fs on read failure + v2.51.24
chore: regenerate Cargo.lock for v2.51.23
fix(sync): prevent subscription_guard deadlock when receivers dropped + v2.51.23
chore: regenerate Cargo.lock for v2.51.22
fix(perf): restore uring_fs fast path for spool reads + parallel DNS pre-resolve + v2.51.22
chore: regenerate Cargo.lock for v2.51.21
perf(balance): true streaming disk I/O for lol_html link extraction + clippy fixes + v2.51.21
chore: regenerate Cargo.lock for v2.51.20
fix(sync): eliminate ChannelGuard livelock on repeated lock() calls + v2.51.20
chore: regenerate Cargo.lock for v2.51.19
fix(balance): eliminate sync blocking disk I/O from async link extraction + v2.51.19
chore: cargo fmt, clippy fixes, bump all crates to v2.51.18
chore: cargo fmt, clippy fixes, bump all crates to v2.51.18
perf: eliminate UTF-8 round-trips in Chrome content injection pipeline
perf(simd): SIMD UTF-8 validation, zero-copy link streaming, and byte-level optimizations
chore: regenerate Cargo.lock for v2.51.17
test(e2e): Chrome hedge parallel_backends E2E tests + v2.51.17
test(e2e): add Chrome + hedge + parallel_backends E2E tests
chore: regenerate Cargo.lock for v2.51.16
fix(hedge+parallel_backends): collect backend results in hedge Chrome path + atomic EMA updates + v2.51.16
perf(balance): pressure-aware spool + lock-free vitals for scaling + v2.51.15
fix(parallel_backends): add backend_timeout_ms to test constructors
fix(parallel_backends): hard outer timeout prevents stalled backends from blocking Chrome + v2.51.12
feat(chrome): multi-endpoint connection failover + v2.51.11
perf(hedge): zero-latency signal-based Chrome hedge escalation + v2.51.10
feat(hedge): Chrome hedge via new WS connection on stall + OOM-safe pre-alloc caps + v2.51.9
perf(async): eliminate all spawn_blocking from spool I/O + async DNS via hickory-resolver + v2.51.8
perf(balance): tokio mpsc cleanup — zero-CPU idle, instant wake + v2.51.7
perf(balance): raise per-page spool threshold to 3 MiB, memory-first design + v2.51.6
perf(balance): batch-drain cleanup task, reduce idle polling latency + v2.51.5
fix(balance): remove thread::sleep from flush_cleanup, use yield_now + deadline + v2.51.4
perf(balance): tokio-native spool cleanup task + v2.51.3
perf(balance): dedicated cleanup thread for spool file deletion + v2.51.2
fix(balance): binary_file flag in transformations, clippy fixes + v2.51.1
feat(page): add binary_file flag for disk-safe binary detection + v2.51.0
fix(balance): make shed_page_html fully async + v2.50.9
refactor(balance): transform_content_send uses byte pipeline by default, clean up dead code + v2.50.8
feat(balance): public async content API — get_html_async, stream_html_bytes_async + v2.50.7
perf(balance): fully async spool I/O, simplified decision logic + v2.50.6
perf(balance): async disk I/O for crawl paths, eliminate runtime blocking + v2.50.5
perf(balance): Arc-shared spool files, cached decisions, EMA adaptive thresholds + v2.50.4
test(balance): add subscription e2e test for disk streaming + v2.50.3
fix(balance): gate spool code behind not(decentralized) + v2.50.2
chore(balance): add unconditional is_html_on_disk() API + v2.50.1
feat(balance): disk-backed HTML spool for memory-balanced crawling + v2.50.0
perf(parallel_backends): drop backend JoinSet immediately after race to free losing pages + v2.49.1
feat(adblock): add adblock_easylist feature, bump all crates to 2.49.0
perf(spider): automatic proxy-host DNS caching with adaptive refresh
perf: skip HTML link extraction for single-page crawls + v2.48.52
chore: remove lightpanda feature flag from parallel_backends + v2.48.51
fix: filter honeypot links (aria-hidden) in all CSS selectors + v2.48.50
perf: zero-alloc SIMD case-folding hash via case_insensitive_string 0.2.11 + v2.48.48
perf: single-pass domain extraction, faster parent_host_match + v2.48.47
perf: zero-copy HTML in all link extraction paths + right-sized buffers + v2.48.46
perf: zero-copy XML stripping, right-sized response buffers + v2.48.45
perf: eliminate String allocation per link in push_link hot path + v2.48.44
fix: memory safeguards — bounded extra_links, eliminate page clones in channel sends + v2.48.43
fix: add memory pressure safeguards to parallel_backends + v2.48.42
fix: DNS errors (525) no longer trigger should_retry in get_error_status_base + v2.48.41
fix: move crawl delay to only apply before HTTP requests, not every loop iteration (#383) + v2.48.40
chore: bump v2.48.39, downgrade CDP timeout log to debug
fix: remove stray cache_namespace arg from run_remote_multimodal_extraction call + v2.48.38
fix: add missing cache_namespace param in smart feature call sites + v2.48.37
feat: configurable cache namespace for partitioning cache keys + v2.48.36
fix: add missing content_map field to decentralized Page struct + v2.48.35
feat: multi return_format support for spider cloud + v2.48.34
feat: SpiderCloudReturnFormat enum for type-safe return_format + v2.48.33
docs: improve README examples with limit param and structured output + v2.48.32
feat: add page.get_content() + spider_cloud_markdown example + v2.48.32
fix: remove Option from subscribe, match dep versions, fix all-features compile + v2.48.31
fix: remove Option from subscribe, always compile + v2.48.30
docs: add CLAUDE.md development guide with Spider Cloud configs, examples, and testing
fix: enable doc_cfg for docs.rs builds, clean up README + v2.48.29
simplify README, placeholder API key safeguard + v2.48.25
fix: ungate use url::Url for decentralized feature + v2.48.24
perf: pre-compile lol_html CSS selectors via OnceLock + v2.48.23
perf: zero-alloc per-link From<&str>, cache drain mem::take + v2.48.22
perf: arena/buffer reuse — zero-alloc CaseInsensitiveString, EMA HashSets, XML buf pool + v2.48.21
perf: add mimalloc allocator option, allocator benchmarks, README guidance
perf: sync CPU/memory state in get_semaphore, pre-allocate link HashSets + v2.48.20
perf: fix last fake-async rewriter in utils/mod.rs, add cooperative yield + v2.48.19
perf: eliminate per-link clone in insert_link — accept &ref, clone only on insert + v2.48.18
perf: eliminate fake-async in crawl loops, sitemap parsing, CSS selectors + v2.48.17
perf: lock-free cache, sync memory state, cooperative rewriter yields + v2.48.16
fix: smart mode upgrade heuristics — weighted scoring, tracker filtering, anti-bot scan + v2.48.15
feat: CLI authenticate command for Spider Cloud API key + v2.48.14
fix: docs.rs build failure — explicit feature list + v2.48.12
fix: docs.rs build failure — replace all-features with explicit feature list
fix: retry logic hardening — proxy-aware 401, full jitter backoff, stale page breaks + v2.48.11
test: comprehensive error handling and retry logic validation
fix: MCP scrape use with_limit(1) to prevent crawling (#380) + v2.48.10
fix: parallel backends CDP/WebDriver run without lightpanda/servo features + v2.48.9
fix: page.html private field + unused_mut warning + v2.48.8 release
fix: suppress dead_code warnings in spider_mcp state
chore: update Cargo.lock for v2.48.7
chore: bump all crates to v2.48.7
feat: per-backend proxy support for parallel backends
feat: parallel backends memory optimizations + v2.48.6 release
fix: release.sh cd always returns to root on publish failure
feat: spider_mcp Spider Cloud support via SPIDER_API_KEY
chore: align internal dep versions for v2.48.5 release
fix: run remote cache before parallel backends in Chrome path
docs: add spider_mcp to README
chore: update Cargo.lock for v2.48.4
feat: add spider_mcp crate — MCP server for Spider web crawler
chore: update Cargo.lock for v2.48.3
fix: parallel backends probe-on-first-request + connect timeout
fix: restore full setup_chrome_events for LightPanda
feat: full Chrome config parity for LightPanda backend
chore: update Cargo.lock for v2.48.2
chore: clippy fixes + bump crates to v2.48.2
perf: pass full handler config to LightPanda via connect_with_config
chore: update Cargo.lock for v2.48.1
chore: bump crates to v2.48.1
feat: randomise launch order between primary and backends
feat: add parallel crawl backends — race LightPanda/Servo alongside primary (v2.48.0)
style: fix formatting in website.rs cfg attribute
feat(cli): add --return-format flag for content transformation (v2.47.93)
fix: gate unused variables behind cache feature cfg
chore: update Cargo.lock for v2.47.92
feat(cli): add --warc flag for WARC output (v2.47.92)
style: fix formatting in warc_chrome example
docs(examples): add WARC output examples for HTTP and Chrome modes
chore: bump all crates to v2.47.91
feat(crawl): add lock-free WARC 1.1 output for web archiving (v2.47.91)
feat(crawl): add auto-throttle and ETag conditional caching (v2.47.90)
fix(crawl): fix compile error in crawl_concurrent_raw - clone antibot patterns before spawn
feat(crawl): user-configurable antibot detection patterns via CustomAntibotPatterns
feat(crawl): expand antibot detection — 7 new providers, server header scanning, 16 new patterns
fix(crawl): restore 6 antibot body patterns accidentally dropped in v2.47.85
perf(crawl): optimize antibot detection — use LeftmostFirst + find() instead of iterator
chore: bump all crates to v2.47.84
fix(crawl): expand anti-bot detection patterns for Cloudflare, Akamai, PerimeterX, DDoS-Guard
chore: bump all crates to v2.47.83
fix: resolve cargo fmt and clippy errors across workspace
fix(crawl): retry antibot 403s instead of treating as hard blocks
chore: bump all crates to v2.47.81
fix(spider): reclassify Chrome error pages as 599, trigger retry
fix(crawl): detect Chrome error pages (ERR_TUNNEL_CONNECTION_FAILED etc.) as invalid content, bump v2.47.80
fix(audit): ignore RUSTSEC-2026-0049 rustls-webpki CRL advisory
fix(crawl): add subscription_guard to all single-page fast paths, bump v2.47.79
chore(deps): bump all crates to v2.47.78
style: cargo fmt spider_worker
fix(deps): upgrade warp 0.3→0.4 to resolve RUSTSEC-2026-0049
style: cargo fmt
perf(cache): optimize SmartCache, SelectorCache, and cache read/write paths, bump v2.47.77
chore(deps): bump chromey v2.42.5, update lockfile, bump v2.47.76
style: cargo fmt
docs(spider): add doc(cfg) feature badges for traits, enable docs.rs all-features, bump v2.47.75
fix(chrome): skip redirect-caused net::ERR_ABORTED instead of aborting navigation
feat(spider): add PageData, Crawler trait abstractions, bump v2.47.73
perf(dns_cache): zero-alloc cache hits via Arc<[SocketAddr]> + index iterator, bump v2.47.72
perf(spider): skip robots.txt for single-page, enable TCP keepalive always, wire DNS cache into reqwest resolver, bump v2.47.71
chore(deps): bump strum 0.28, phf 0.13, gemini-rust 1.7, chromey 2, llm_models_spider 0.1, bump v2.47.70
perf(cache): batch LRU eviction to reduce write-lock hold time, fix aws-lc-sys audit, bump v2.47.69
fix(spider): add decentralized stubs for set_url_parsed_direct and links_full, bump v2.47.68
perf(spider): remove unnecessary Box allocations and redundant string conversions, bump v2.47.67
feat(cache): dump chrome responses to remote cache server
fix(spider): fall back to HTTP crawl when Chrome is unavailable (#373), bump v2.47.65
feat(spider): proxy support for LLM HTTP requests (#378), bump v2.47.64
test(spider): timing tests for cache_mem + chrome_remote_cache skip_browser, bump v2.47.63
perf(spider): cache_mem auto skip_browser for chrome_remote_cache too, bump v2.47.62
perf(spider): cache_mem auto-enables skip_browser for Chrome mode, bump v2.47.61
fix(spider): deterministic CDP event listener shutdown via watch channel, bump v2.47.60
feat(spider): Chrome mode remote_addr from CDP Network.responseReceived, bump v2.47.59
fix(spider): add 3s timeout to remote cache fallback in chrome_remote_cache-only path, bump v2.47.58
fix(spider): remote cache fallback for Chrome mode when cache_mem enabled, bump v2.47.57
perf(spider): trie Box keys + manual byte-walk + memchr dot scan, bump v2.47.55
perf(spider): SIMD-accelerated byte scanning via memchr, u64 word comparisons, unrolled FNV hash, bump v2.47.54
perf(spider): bloom filter bitmask addressing + inline early-exit, interner zero-clone extend, bump v2.47.53
perf(spider): eliminate hot-path allocations — push_link, attr removal, normalize_html, bump v2.47.52
perf(spider): NUMA thread pinning + zerocopy wire parsing, bump v2.47.51
perf(spider): bytes::Bytes zero-copy page passing, mmap+hugepages bloom filter dedup, bump v2.47.50
perf(spider): data structure optimizations — O(1) frontier pop, zero-alloc header matching, eliminate HashSet clones, bump v2.47.49
feat(spider_agent): complete RemoteMultimodalConfigs builder pattern (#376), bump v2.47.48
feat(spider_cli): add --chrome-connection-url, --cookie, --stealth CLI flags, bump v2.47.47
perf(spider): Chrome request reliability — min 2 retries, backoff, fresh tabs, browser health signal, bump v2.47.46
fix(spider): Chrome status code defaults to 599 instead of 200 on no response, bump v2.47.45
feat(spider): Spider Browser Cloud integration via wss://browser.spider.cloud, bump v2.47.44
fix(spider): fix clippy redundant_guards warning in get_html_cow, bump v2.47.43
perf(spider): memory optimizations — Cow HTML, bounded caches, tuned buffers, bump v2.47.43
perf(spider): eliminate redundant allocations in crawl hot paths, bump v2.47.42
feat(spider): adaptive hedge tracking for HTTP paths, bump v2.47.41
feat(spider): variance-aware adaptive hedge delay with auto-tuning signals, bump v2.47.40
fix(spider): fix Chrome hedge never firing on slow sites, use raw EMA for adaptive delay, bump v2.47.39
fix(spider): add backoff for 599/598 proxy errors in get_timeout(), bump v2.47.38
feat(spider): add Chrome hedged requests with adaptive delay tracking, bump v2.47.37
fix: configurable automation timeout, uring_fs test deadlock, unused import (#374), bump v2.47.36
fix: clippy unnecessary casts and hex literal grouping in splice, bump v2.47.35
chore: bump v2.47.34
feat(spider): add tcp_fastopen, splice, io_uring send/recv, bump v2.47.33
revert(spider): remove needs_retry() from initial_html_length check, bump v2.47.32
fix(spider): set initial_html_length to 0 when page needs retry, bump v2.47.31
fix(spider): set initial_html_length to 0 for empty pages, bump v2.47.30
fix: retry on empty content (200 + no body) and truncated responses
fix: remove useless Into::into after double-box removal (clippy)
feat: enable dns_cache, rate_limit, request_coalesce in __basic defaults
feat: 10 performance & reliability enhancements, bump to v2.47.28
fix: ensure 599/598 errors always set should_retry for website-level retry logic
fix: inline let binding to satisfy clippy let_and_return lint
fix: remove retry suppression for request errors, add status-based retry classification
feat: async file I/O for TempStorage + ExperienceMemory, bump to v2.47.25
chore: bump all crates to v2.47.24
style: cargo fmt
fix: remove unnecessary i32 cast on libc::EINPROGRESS
feat: io_uring TCP connect + lightweight background runtime, bump to v2.47.23
fix: replace tokio-uring with raw io-uring crate + kernel probe, bump to v2.47.22
fix: eliminate panic sites and unsafe indexing, bump to v2.47.21
fix: detect and flag truncated responses from upstream Content-Length mismatch
chore: bump all crates to 2.47.18
fix: enforce hard wall-clock timeout on all crawl entry points
feat: Auto API mode — api.openai.com uses Responses API, others use Completions, bump to v2.47.17
feat: OpenAI Responses API support (opt-in), bump to v2.47.16
feat: expose underlying HTTP client and configurable timeout on agent, bump to v2.47.15
fix: auto-append /chat/completions for OpenAI-compatible URLs, add proxy support to agent, bump to v2.47.14
fix: DNS resolve errors (525) skip all retry loops, bump to v2.47.13
chore(crawler): add v2.47.12
fix: 599 status with real content no longer misclassified as ConnectError, bump to v2.47.11
fix: include webdriver feature Box<[u8]> conversion, bump to v2.47.10
perf: reduce Page html from Box<Vec> to Box<[u8]>, bump to v2.47.9
perf: unbox redirect_limit/cookie_str/request_timeout, remove charset alloc, bump to v2.47.8
perf: Arc for external_domains_caseless, skip charset lowercase alloc, bump to v2.47.7
perf: eliminate wasteful allocations on hot paths, bump to v2.47.6
fix: lazy anti-bot body check in smart mode, exhaustive pattern tests, bump to v2.47.5
feat: detect Alibaba TMD anti-bot, force Chrome upgrade in smart mode, bump to v2.47.4
fix: validate 598/599 as genuine errors before ConnectError, bump to v2.47.3
fix: 598/599 status codes return ConnectError, bump all crates to v2.47.2
feat(chrome): add network idle timeout to prevent tarpit during CDP navigation
feat: add per-chunk idle timeout to prevent tarpit/slow-drip antibot attacks
chore: bump spider_agent to v2.46.3 (v2.46.2 already published)
chore: bump all crates to v2.46.2
feat(agent): retry on different model for 502/503/429/5xx errors
feat(agent): add Anthropic Messages API thinking/extended thinking support
style: fix rustfmt formatting in reqwest 0.13 upgrade
feat: upgrade to reqwest 0.13 ecosystem
feat(agent): eliminate panics, silent failures, and unbounded growth
feat(agent): action handler hardening — duration caps, JS escape, timeout guards
feat(agent): eval timeouts, spawn concurrency limiter, PageWaitStrategy, panic hardening
feat(agent): action feedback, partial result preservation, and JS error capture
style: fix rustfmt formatting in smart_vs_chrome tests
chore(release): 2.45.25
fix: use final redirect URL as base for relative link resolution (#364)
chore: ignore transitive audit advisories with no direct fix
chore: ignore RUSTSEC-2025-0134 (rustls-pemfile via warp)
chore: ignore RUSTSEC-2026-0002 (lru 0.13.0 via wreq)
docs: backfill CHANGELOG v2.45.1-v2.45.22 and add .editorconfig
chore: remove cargo dependabot, keep github-actions only
chore: fix clippy warnings and formatting across workspace
chore(deps): bump flexbuffers 2->25, async-openai 0.32->0.33, clap 4.5.59->4.5.60, actions
docs: rewrite README, add issue/PR templates, security policy, CI improvements
docs: add Contributor Covenant v2.1 Code of Conduct
docs: rewrite CONTRIBUTING.md and add zero-config quick start to README
fix(cache): proper HTTP staleness for Chrome-cached pages
fix(cache): Period policy bypasses HTTP is_stale for Chrome-rendered pages
chore: bump all workspace crates to v2.45.22
fix(cache): enable cache_chrome_hybrid_mem feature for Chrome cache writes
chore: bump all crates to 2.45.21, chromey 2.38.4
chore(agent): fix model router import
docs(spider_agent): add SKILLS.md for engine skill integration reference
chore(chrome): 2.45.19 root sandbox default launch
fix: add no_sandbox() to headless BrowserConfig (#354)
test: comprehensive crawler-test.com integration suite (302 tests, 408 URLs)
feat: per-round model pool routing for cost-optimized automation + bump 2.45.18
perf: skip model scoring for pools ≤ 2 models + bump 2.45.17
test: comprehensive multi-LLM router reliability tests + bump 2.45.16
release(cli): 2.45.15 wait_for
feat(cli): add wait-for capabilities to spider_cli (#352)
fix: Chrome mode honors wait_for config for networkIdle before HTML extraction
fix: smart mode lifecycle waiting — match Chrome coverage without re-fetch
fix: auto-retry www. URLs on SSL protocol errors — strip www prefix
fix: reject empty HTML from all cache and seeded resource paths
fix: invalidate empty HTML shell responses on cache read path
feat: deferred Chrome — cache-only crawl phase before browser launch
feat: cache-first fast path — skip browser/HTTP when cache has data
fix: add Website::with_hedge() forwarding method + bump to v2.45.7
feat: add work-stealing (hedged requests) for slow crawl requests
fix: skip HTML-specific heuristics in cache empty check for non-HTML content
chore: bump all workspace crates to v2.45.4
fix: skip caching empty/near-empty HTML responses
chore: bump all workspace crates to v2.45.3
chore: bump spider to v2.45.3
fix(uring_fs): fix compile errors when io_uring feature is enabled
perf(io_uring): add StreamingWriter for streaming file I/O, bump v2.45.2
feat(io_uring): expand io_uring integration for file I/O, bump v2.45.1
refactor(agent): extract types and HTML cleaning into spider_agent_types and spider_agent_html crates
chore(deps): major version bumps, bump v2.45.0
chore: bump v2.44.42
perf(website): use take() instead of clone() for subdomain base URL
fix(website): add tests for subdomain relative URL resolution, bump v2.44.41
fix(website): use page's own URL for relative link resolution on subdomains
chore: clarify network event collection comment (#350)
feat(agent): integrate llm_models_spider v0.1.9 with smart model selection
docs: add missing feature flags, Spider Cloud example, and integration docs
refactor(agent): move all skill content to spider_skills crate
feat(agent): add L15-L48 skills, NEST/CIRCLE engine loops, haiku benchmark
fix: add should_use_chrome_ai to stub RemoteMultimodalConfigs
feat(agent): add L8-L14 skills, WAM engine loop, and Chrome AI improvements
chore(agent): add improved steps
test(website): fix rotation cases
feat(agent): optimize text-only rounds and bump workspace crates
chore: fix lint warnings and feature-combo build regressions
fix: improve agent concurrency and ensure all-feature builds
feat(cache): optimize automation caching and bump crates to 2.44.33
Add crawl_smart cache parity test and bump crates to 2.44.32
Improve cache skip-browser flows and add remote cache e2e example
feat: add spider cloud end-to-end examples and bump crates to 2.44.30
Update Cargo.lock for 2.44.29
Improve remote multimodal automation reliability and release 2.44.29
Expose optional automation reasoning in metadata and bump 2.44.28
release: bump workspace crates to 2.44.27
feat(spider_cli): add runtime http/headless mode controls and release 2.44.26
chore(client): add fallback agent declaring
docs: update dual-model example with per-endpoint URL and API key configuration
feat: dual-model routing examples and comprehensive tests
feat: extraction-only mode optimization for single-round data extraction
docs: add Spider Cloud reliability integration section to README
fix: feature flag compilation issues across multiple feature combos
test(ignore): add test_crawl_subdomains to ignore
fix: feature flag compilation issues across wreq, agent_full, and cache combos
chore: bump to v2.44.20 for release
fix: broken chrome feature — missing relevant field and cfg gate on detect_cf_turnstyle
chore: bump to v2.44.19 for release
feat: integrate spider_skills crate for skill types and web challenge definitions
feat(agent): add URL-level relevance pre-filter for crawling, bump to v2.44.18
perf: fast-path trie protocol detection, remove redundant char boundary check
perf: optimize trie, robot parser, and URL preparation hot paths, bump to v2.44.17
feat(agent): add relevance gate for remote multimodal crawling
perf: optimize core pipeline hot paths, add benchmarks, bump to v2.44.15
chore: bump versions to 2.44.14
fix(agent): use CDP mouse methods for all click actions to maintain position tracking
docs: simplify root README intro
docs: add spider_cloud to root README, improve API response handling
chore: bump versions to 2.44.13
feat(spider): add spider_cloud integration and S3 skills loading
chore: bump versions to 2.44.12
feat(spider_agent): add dual-model routing and long-term experience memory
fix(spider): fix broken glob feature compilation and logic errors
chore: bump versions to 2.44.10
feat(spider_agent): improve skill triggers and board reading for web challenges
feat(spider_agent): improve skill strategies for real browser interaction
feat: sync skill content with improved efficiency directives
chore: bump versions to 2.44.9
feat(spider_agent): add Claude-optimized automation features
chore: bump versions to 2.44.8
feat(spider_agent): add concurrent page spawning with OpenPage action
fix(spider_agent): update system_prompt_compiled test for locked prompt
chore: bump versions to 2.44.7
feat(spider_agent): integrate llm_models_spider for auto-updated vision detection
feat(spider_agent): add inline vision model detection
chore: bump versions to 2.44.6
feat(spider): enable HTTP extraction without agent_chrome feature
feat(agent): enhance CAPTCHA handling and lock system prompt
feat(agent): improve web automation for CAPTCHAs and visual challenges
docs(readme): add spider_agent section and link
chore(examples): add remote_multimodal_multi example
feat(agent): consolidate automation into spider_agent with seamless feature integration
chore(example): add extraction example multi
chore(website): fix scrape subscription hang
chore(examples): add remote_multimodal
chore: bump all crates to v2.44.3 for version parity
fix(spider): keep spider_agent optional, add agent_webdriver feature
feat(spider): make spider_agent a required dependency
feat: granular usage tracking in AutomationUsage
chore(spider): bump to v2.43.23, update spider_agent to 0.6
feat(spider_agent): add usage limits, custom tools, and granular tracking
feat(automation): add api_calls tracking and HTTP extraction support
chore(spider): bump to v2.43.21, update spider_agent to 0.5
feat(spider_agent): add complete WebAutomation parity and PromptUrlGate
docs: expand v2 changelog with comprehensive feature documentation
docs: update changelog for v2.43.20 and spider_agent releases
chore(spider): update spider_agent dependency to 0.4
feat(spider_agent): add performance optimizations for automation
feat(spider_agent): add comprehensive automation module
chore(spider_agent): bump version to 0.2.0
feat(spider_agent): enhance memory with URL/action/extraction history
feat(spider_agent): add webdriver support via thirtyfour
feat(spider_agent): add chrome browser and temp storage support
feat(spider_agent): add usage tracking and multimodal example
fix(spider): fix doctest and update chromey for adblock compatibility
feat(spider_agent): add standalone multimodal agent crate
fix(search): use reqwest::Client directly for cache feature compatibility
chore(docs): fix rustdoc broken intra-doc links
feat(search): add search_and_extract and research methods
feat(search): add web search integration with multiple providers
perf(automation): byte-size-based smart cleaning for best performance
feat(cache): add skip browser mode + smart HTML cleaning
fix(automation): correct lol_html API usage and add intelligent screenshot detection
feat(automation): add Phase 3 agentic features - autonomous agent, action chaining, error recovery
feat(automation): add Phase 2 agentic features - selector cache, structured outputs, map API
feat(automation): add simplified agentic APIs - act(), observe(), extract()
feat(automation): add agentic memory for multi-round automation
feat(automation): add prompt-based website configuration
perf(website): remove unnecessary clones and allocations
test(automation): add tests for JSON extraction from thinking models
fix(automation): improve JSON extraction with proper brace matching
feat(automation): add llm_json for robust JSON parsing from LLM outputs
feat(automation): add HTTP extraction support without chrome feature
fix(automation): improve best_effort_parse_json_object parsing
fix: extract JSON from LLM responses with reasoning text
feat(automation): add ExtractionSchema for structured extraction output
fix(page): add remote_multimodal fields to decentralized Page struct
feat(automation): add token usage tracking and extraction support for RemoteMultimodalEngine
chore: update to spider_fingerprint 2.38.0 from crates.io
feat(webdriver): integrate spider_fingerprint stealth spoofs
docs: refresh README with streaming, Chrome, and WebDriver examples
feat(webdriver): add automation script support
chore(spider): bump version to 2.41.0
feat(spider): add WebDriver support via thirtyfour crate
test(page): add feature flag test coverage
fix(website): gate seeded page link extraction for decentralized feature
chore(spider): bump version to 2.40.4
fix(website): extract links and metadata from seeded pages
fix(page): fix metadata extraction in streaming functions
fix(page): add metadata extraction for HTTP requests
feat(chrome): add remote multi_modal web automation
chore(page): add missing gemini
chore(page): add forbidden validation retry
chore(page): add set_duration_elasped
chore(solvers): add lemin
chore(website): add warm model
feat(solver): add built in gemini nano support
chore(mod): add gemini solve
chore(smart): add handles
chore(smart): add imperva handle
chore(smart): add sessionStorage detection
release(minor): 2.39.0
Add with_on_should_crawl_callback_closure (#346)
chore(website): add modify http client builder
chore(website): add with_referrer builder
chore(chrome): add extra cleanup caps
chore(mod): fix request error compile
chore(website): add hint setters
Fix breakage on openai example (#345)
chore(chrome): add header cleanup
chore(smart): add cypher mismatch check retry
chore(cookies): add cookie jar re-use
chore(chrome): fix bytes calculation smart rendering
chore(website): add extra_information flag
feat(gemini): add Gemini AI support for dynamic browser scripting (#344)
chore(website): add extra info storing
feat(website): add cmd crawling
fix(chrome): add automatic chrome executable detection (#343)
chore(network): add blacklist patterns
chore(page): fix smart rendering
chore(website): add pub buildable crawler
chore(website): add copy crawl status
chore(smart): add xml request detection
chore(chrome): add whitelist patterns
chore(chrome): add click automations
chore(page): add empty trim check
chore(validation): add to list
chore(page): add false validation
chore(page): add template false 403
chore(website): add initial bytes tracking
chore(page): add empty html
chore(chrome): add emulation check
chore(chrome): add cache cfg
chore(smart): add react app ssr check
chore(smart): fix empty page assign
chore(page): add common type list ignore
chore(smart): add hydration list
chore(smart): add pierced nodes selecting frames
chore(page): add gateway timeout
chore(timeout): add gateway delay
chore(page): add page empty fix
chore(smart): add nuxt detection
chore(smart): fix redirect assign
chore(smart): fix page rendering on re-render
chore(smart): revert watch methods
chore(chrome): fix rewrite content base tag
chore(smart): fix script tag removal
chore(smart): fix redirect page content
chore(configuration): fix with_wait_for_idle_network0 and with_wait_for_almost_idle_network0 builder
chore(cache): add waf url check
chore(real_browser): fix cf detection
chore(page): fix http network error status code
chore(chrome): add initial document status code fix
chore(chrome): fix compile remote cache
Fix "real_browser" disabled (#336)
chore(cache): fix cache_options disk flag [#334]
feat(chrome): add remote cache support
chore(website): add connect error status
chore(crate): add chrome_serde_stacker flag
chore(website): fix chrome initial crawl status
feat(chrome): add idle_network0 almost_idle_network0 wait for
perf(chrome): add command send only closing chrome
chore(chrome): add default ua & ref launch chrome headers
chore(chrome,cache): add time check retrieval
chore(website): add dynamic link find
chore(website): add timeout builder methods
chore(chrome): remove extra client header
chore(chrome): fix cache compile
chore(chrome): fix cache options compile
chore(chrome): add shared hybrid cache retrieval
chore(cli): add proxy_url [#330]
chore(chrome): fix handler config [#329]
chore(chrome): add event data handling internally
chore(chrome): add wire bytes blocking
chore(cli): fix download unicode handling [#325]
chore(chrome): add stop_loading byte limit
chore(chrome): fix compile [#328]
chore(parser): add base tag respecting [#323]
spider_cli: fix duplicated argument -r (#324)
chore(chrome): fix max page bytes cancelling
chore(page): fix invalid dns retry handling
chore(chrome): add type automation
chore(website): add get_extra_links
feat(disk): add shared_state multi profiling
chore(chrome): add click point
chore(chrome): fix guard lock [#315]
chore(chrome): fix clickable selector default
chore(page): fix title metadata extracting
chore(tracker): add event metadata
chore(utils): fix duration elasped handling http [#306]
chore(chrome): fix compile auto geolocation [#311]
chore(website): fix platform interface binding [#309]
chore(time): fix page duration tracking [#304]
chore(utils): fix protocol handling invalid utf8
chore(crate): fix compile spider_fingerprint
chore(builder): add interface and local_address binding
chore(cli): add status_code and remote_addr flags [#303]
chore(cli): add time feature flag [#302]
chore(real_browser): add simple click protected
chore(sitemap): add xml check pre-parse
chore(crate): bump quick-xml@0.38
chore(cli): add return_headers [#301]
chore(cli): add full-resources arg
chore(utils): remove unused BufMut
chore(utils): fix fetch advance out of bounds panic
chore(utils): fix bufmut capacity panic
chore(website): add csp enable/disable builder
chore(chrome): add cdp handling cf
feat(chrome): add auto_geo detection
chore(emulation): add cdp set hardware_concurrency
chore(emulation): add back userAgentData emulation
chore(docs): fix spider_utils documentation crate
chore(crate): cleanup crate purpose
chore(chrome): add click_and_drag
chore(docs): update relevant info
chore(chrome): add full user_agent_metadata handling
chore(chrome): add drag, double_click, and click_modifiers
chore(chrome): fix firewall flags compile
chore(chrome): re-export firewall-rustls
chore(chrome): re-export firewall
chore(fingerprint): add to gpu list
chore(chrome): add navigator.platform emulating
feat(chrome): add max_page_bytes control
chore(chrome): add default flag serde0 pdl
chore(chrome): add backwards compat scroll_by
chore(chrome): fix delta for mousewheel
chore(chrome): add to blocklist
chore(chrome): add to blocklist
chore(spider_chrome): add set_extra_headers and screencast
chore(spider_chrome): add page set headers
chore(spider_chrome): add rust-tls flag
chore(fingerprint): fix firefox spoof
chore(fingerprint): allow reference viewport
chore(chrome): add scroll method
chore(fingerprint): fix referer defaulting
chore(fingerprint): fix empty agent handling
chore(fingerprint): fix referrer header
chore(fingerprint): add header detail control
chore(fingerprint): add public title_case_header
feat(fingerprint): add emulate_headers
chore(fingerprint): remove openssl build defaults
chore(fingerprint): add optional serde compile
chore(fingerprint): add __pwInitScripts optional spoof
feat(fingerprint): add centralized emulate
chore(fingerprint): fix hardwareConcurrency name
chore(fingerprint): fix hardwareConcurrency worker handling
chore(fingerprint): add ua_full_version automation
chore(fingerprint): fix chrome spoof
chore(docs): fix chromiumoxide fork intro docs
chore(fingerprint): add navigator.hardwareConcurrency spoof
chore(examples): fix anti_bot with_user_agent
chore(fingerprint): fix device_pixel_ratio mac defaults
chore(fingerprint): fix hide chrome
chore(fingerprint): prep fingerprint canvas noise
chore(fingerprint): add profiles start
chore(fingerprint): add env section
chore(fingerprint): fix userAgentData getHighEntropyValues
chore(chrome): add to block list
chore(fingerprint): fix webgl worker spoof gpu
chore(chrome): default enable service_workers
chore(fingerprint): fix navigator.userAgentData prototype
chore(fingerprint): fix plugins spoof toJSON
chore(fingerprint): revert plugins
chore(fingerprint): fix plugins spoof
chore(fingerprint): add touchscreen spoof
chore(chrome): add to block list
chore(fingerprint): fix screen spoof color_depth
chore(spoof): add plugins JSON.stringify fix
chore(fingerprint): add spoof media labels
chore(chrome): add thread rng re-use
chore(chrome): add to block list
chore(chrome): add to block list
chore(docs): fix spoof_ref rustdoc
chore(fingerprint): add re-export spider_fingerprint
chore(fingerprint): add spider_fingerprint workspace
chore(chrome): add permissions and notifications spoof
chore(spoof): fix rust doc spoof_referrer_google
chore(chrome): add spoof media codecs
chore(chrome): add spoof screen
chore(real_browser): add header merging
chore(chrome): add to block list
chore(crate): add __basic flag no tls [#297
chore(crate): add http2 feature flag reqwest [#297]
chrome_remote.rs: Add note about headless-browser (#296)
chore(docs): add chrome-headless-shell recommendation
chore(chrome): bump cdp revision 1457408
chore(chrome): update browser,js pdl files
chore(page): fix returning content on errors
chore(chrome): remove unused cfg-if
chore(chrome): add to block list
chore(website): add initial_waf_detected and initial_page_retried
chore(website): add set_status
chore(chrome): add to block list
chore(crate): bump sonic-rs@0.5
chore(crate): bump lock backtrace@0.3.75
chore(chrome): add browser dispose on end
chore(chrome): add mobile chrome headers
chore(chrome): fix concurrent context creation
chore(chrome): fix create new context
chore(chrome): add frame_id navigate params
chore(page): remove unused og meta
chore(page): add compact_str metadata
chore(serde): add back serde serialize AIResults
feat(page): add metadata extracting
chore(chrome): fix cfg chrome_args
chore(sitemap): add auto sitemap include whitelist
chore(real_browser): add rng init re-use
chore(http): add default error status
chore(chrome): add to block list
chore(chrome): add with_stealth_advanced stealth control
chore(chrome): add webdriver real spoof
chore(chrome): add gpu requestAdapter spoof
chore(chrome): split mac and linux gpu spoof
chore(chrome): add real native userAgentData spoof
release(minor): 2.36.114 serde configuration defaults
Add serde(default) for Configuration. (#290)
chore(chrome): fix plugins script
chore(crawler): add apache,openresty 403 detection
chore(examples): add default fingerprint disable full pass anti-bot
chore(chrome): fix stealth fingering mixing
chore(chrome): fix wait_for_dom
chore(chrome): add spoof fingerprint gpu with_fingerprint_advanced
chore(openai): box gpt configs
chore(openai): fix run openai call
chore(chrome): fix sitemap crawling xml files
chore(chrome): prep gpu fingerprint disable
chore(chrome): add plugins spoof entry
chore(chrome): fix missing bitness
chore(website): add requires_javascript detection
chore(disk): add optional sqlite builder method
chore(chrome): add to block list
chore(stealth): fix userAgentData setting
chore(crawler): add headers clear method
chore(chrome): fix turnstile detection
chore(chrome): fix pdfViewerEnabled stealth
chore(adblock): fix adblock compile [#288]
chore(chrome): fix fingerprint script
chore(chrome): fix header casing
chore(chrome): fix stealth mode initial webdriver set
chore(chrome): add spoof navigator userAgent high entro
chore(website): add with_dismiss_dialogs builder method
chore(page): fix duration elasped default compile
chore(chrome): disable runtime default
chore(chrome): fix viewport handling
chore(sitemap): fix budget tracking
chore(page): add 404 status parsing
chore(spoof): fix trailing slash ref
chore(client): fix aggresive keep-alive
chore(website): add get_initial_anti_bot_tech
feat(antibot): add anti-bot detection
chore(chrome): add anti-bot detection
chore(xml,page): fix xml parsing html
chore(chrome): add unique isolated world generation
chore(chrome,sitemap): fix xml parsing sitemap
chore(sitemap): fix crawling sitemap return html response
chore(cookies): fix duplicate cookie provider setup
chore(headers): add viewport width
chore(real_browser): add header preserving
chore(website): fix client host header
chore(decentralized): fix compile set_url_parsed_direct_empty
chore(chrome,sitemap): add chrome rendering sitemap
chore(page): fix error status code handling
chore(website): fix xml parsing links [#284]
release: 2.36.61
fix: errors with page.authenticate (#286)
chore(chrome): fix sitemap xml parsing content
chore(chrome): revert enable stealth with agent
chore(chrome): fix custom event paused event listener
chore(chrome): fix cache disable duplicate
chore(real_browser): fix arch and model headers
chore(website): fix http dup ua
chore(headers): fix header casing http
chore(chrome): fix unknown status code security tracking
chore(chrome): fix gpu spoof
chore(chrome): fix spoof webdriver linux
chore(chrome): fix spoof runtime
chore(real_browser): add sec bitness,form,scheme default headers
chore(chrome): fix invalid redirects
chore(real_browser): add missing Priority header
chore(chrome): add rtf file handling
chore(real_browser): add linux fp target
chore(real_browser): fix linux default headers
chore(chrome): fix default accept lang
chore(chrome,headers): fix compile store headers
chore(chrome): add missing linux headers
chore(chrome): fix default fp overwrites
chore(chrome): fix wait for dom exit
chore(chrome): fix compile fp
chore(chrome): add platform fp handling
chore(chrome): fix casing headers raw real_browser
chore(chrome): add to block list
chore(chrome): add dynamic header preserve order
chore(chrome): fix hybrid cache storing valid headers
chore(page): fix invalid page handling content bytes
chore(page): fix document status code handling
chore(real_browser): add accept-encoding client-hint
chore(client): revert ref redirect
chore(real_browser): fix referer header order
chore(real_browser): fix ref header order
chore(website): add firewall bypass
chore(real_browser): fix chrome Sec-Ch-Ua-Platform case
chore(chrome): fix timeout subtracting
chore(chrome): fix request_timeout default
chore(transformations): add <document check
chore(real_browser): add accept-language header
chore(real_browser,chrome): add missing chrome headers
chore(chrome): add real browser loading
chore(chrome): add request timeout bytes ignore
chore(crate): fix default features
chore(decentralized): fix compile event map
chore(chrome): add unknown status code mask
chore(chrome): fix wait_for_dom updating
chore(chrome): fix events 599
chore(chrome): fix sonic-rs utf8 parse
chore(chrome): fix generic unknown status firewall handling
chore(chrome): add buffer timeout cancelled request
chore(chrome): fix content hang
feat(client): add rquest [#280]
chore(chrome): fix queue [#282]
release: 2.34.5
feat(chrome): add support for wss connection (#279)
chore(chrome): add page content asset fallback
chore(chrome): add exact media extraction [#278]
chore(chrome): add masked bytes env
feat(chrome): add req/resp tracker [#254]
chore(chrome): fix asset downloading [#278]
chore(utils): fix DocumentSelectors serde compile error
feat(chrome): add xhr block custom list
chore(lib): re-export spider_firewall
chore(chrome): add to block list
feat(website): add crawl_timeout and with_crawl_timeout builder method
chore(website): add robots parser reference method
chore(chrome): add 598 chrome error
feat(chrome): add ValidateChain web automation
chore(chrome): fix automation scripts wait_for selector
chore(chrome): add to block list
chore(crate): revert fork reqwest-middleware
chore(examples): cleanup mut evalulate
chore(chrome): add to block list [#276]
chore(cache): fix http-cache compile
feat(nix): add default.nix spider,spider_cli [#266]
chore(crate): bump async-openai@0.28
feat(chrome): add asset handling pages [#275]
chore(lib): remove jemalloc from lib
chore(chrome): add ws dispose
feat(chrome): add fetch_chrome_persisted reuse browser
feat(website): add light website.fetch_chrome
chore(control): remove clone on reader
feat(builder): add with_no_control_thread
chore(website): add setup website norobots
feat(send): add crawl raw send
feat(website): add chrome crawl_chrome_send thread safe
chore(chrome): add to block list
chore(decentralized): fix compile bytes
chore(utils): add serde serialize DocumentSelectors
perf(page,chrome): add direct bytes handling
chore(chrome): fix wait_for events sequence
chore(chrome): add navigation network cancel
chore(chrome): fix chrome headed proxy [#272]
chore(budget): fix shutdown
chore(chrome): fix launch timeout
chore(crawl): add shutdown set join_task clear
chore(chrome): add initial page retry
perf(conf): remove box indirection proxies, whitelist, and blacklist
chore(chrome): fix initial page status code set
chore(chrome): add infinite loop document reload protection
chore(chrome): add to block list
chore(chrome): add localhost proxy bypass
chore(chrome): fix firewall compile
chore(chrome): revert dispose quite context
chore(chrome): revert dispose drop
chore(chrome): fix ws handler disconnect dispose
chore(chrome): add handler drop on dispose
chore(sub): fix guard handling
chore(chrome): add handler abort
chore(chrome): fix default sender dispose context message
chore(chrome): fix closing handler drop
chore(chrome): add handler close ws errors
chore(chrome): add web socket error closing
chore(chrome): add about:blank proxy_bypass
chore(chrome): remove loopback local
feat(chrome): add bad website block chrome networking
chore(chrome): add html error extracting
chore(smart): fix retry client errors
perf(smart): add base tag rewrite
perf(bytes): remove BytesMut
chore(chrome): add dispose_on_detach closing
chore(smart): fix missing bytes transferred
feature(smart): add lazy load chrome
chore(website): add reset_status
chore(utils): re-export is_bad_website_url
chore(firewall): add automated list
chore(chrome): add to block list
chore(chrome): bump connection retries
chore(examples): add subscribe_multiple
chore(website): fix page empty status
chore(chrome): add close browser cleanup
chore(website): add new_page direct drop
chore(docs): cleanup firewall readme
feat(firewall): add start of spider_firewall
chore(website): add same tasks joining
chore(chrome): add to block list
chore(website): fix scrape hang [#268]
chore(glob): fix compile [#267]
chore(chrome): add optional cache disable
chore(chrome): add service worker default disable
chore(chrome): add preset init commands
chore(smart): add initial http fallback
chore(chrome): add connection retry ws
release(minor): add cli arg help v2.27.54
chore(cli): trigger help page on missing arguments (#265)
chore(chrome): add to block list
chore(chrome): add info log errors
chore(chrome): add WEBSOCKET_DEFAULTS env variable
chore(website): add direct proxy control
chore(crawler): add force proxy special protocol
chore(proxies): add sock:// ignore http
chore(chrome): add to block list
chore(website): fix disk compile feature
chore(transform): fix default features [#264]
chore(website): fix queue ignore [#256]
chore(chrome): add wait_for check
chore(website): add set_url_only
chore(chrome): add get_cookies browser context id
chore(chrome): revert content handling
chore(chrome): fix outer_html target
chore(request): bump max_duration 5 mins
chore(chrome): fix duration overflow
chore(chrome): add cdp DOM.getOuterHTML
chore(chrome): fix duration page handling timeout
docs: fix unclear documentation around scraping (#262)
chore(chrome): fix wait_for delay ordering
chore(chrome): add to default allowed scripts
chore(page): fix full_resources DOM watch compile [#260]
chore(page): fix full_resources smart compile [#260]
chore(abs): clear query pairs [#257]
chore(chrome): fix rand@0.9 compile
chore(crate): update rand@0.9
chore(chrome): handle socks protocol default
chore(page): return error for status
chore(crate): update cron@0.15
chore(page): cleanup unknown status code usage
chore(chrome): add optional simd flag
chore(chrome): enable tcp keep alive ws connection client
chore(website): fix insert signature compile
chore(page): add port validation links
chore(chrome): update tokio-tungstenite@0.26
perf(chrome): add simd json entry
chore(chrome): add binary message handling
chore(website): add normalize builder method
chore(transformations): update fast_html2md@0.0.44
chore(serde): fix configuration compile cron
chore(page): fix empty base link joining
chore(utils): add back trim on abs path convert
chore(website): fix relative path merge building
chore(website): add normalize handling chrome
chore(website): fix subscribe guard handling queues
chore(utils): add normalizing start
chore(config): set default depth 25
chore(page): remove extra clone external domains
feat(page): add relative directory url handling
chore(page): add status code from error
ci(bench): fix local server install
chore(worker): fix background runtime start
chore(website): fix invalid protocol repair default https
perf(website): add direct page selectors
chore(website): add fast hostname with_external_domains
perf(chrome): add direct about:blank check
chore(website): fix dequeue validation [#251]
chore(website): fix dequeue budget tracking
chore(utils): remove extra check get_path_from_url
chore(website): fix depth validation
chore(page): remove duplicate asset check links validation
chore(website): fix initial q drain [#250]
chore(utils): fix meta content css select
chore(utils): add meta css extracting content
chore(utils): fix meta parsing css extraction
chore(ignore): add xml and yaml ignore
chore(sitemap): add chrome sitemap auto-correct
chore(cache_chrome_hybrid): fix compile flag [#245]
chore(sitemap): add auto find sitemap url path
chore(chrome): add to block list
chore(tests): fix transformations build parse
perf(cli): add page links direct return
chore(page): remove duplicate abs
feat(page): add optional page base passing
chore(page): fix parsing url detection
perf(page): remove parsing url directly
chore(docs): fix spider cargo docs
chore(page): fix build reqwest error_for_status cache flag
chore(page): fix feat decentralize compile error status
chore(page): add new_page_streaming error
feat(page): add page_error_status_details [#244]
chore(connect): fix compile missing libc
chore(license): bump 2025
feat(cargo): add basic feature flag
chore(chrome): fix compile fs flag
chore(page): fix crawling initial redirect
chore(decentralized): fix page blocked_crawl
chore(fs): remove tentril crate
chore(disk): fix cfg aho_corasick
feat(page): add blocked_crawl [#242]
feat(website): add on_should_crawl_callback [#241]
chore(fs): fix feature flag compile fs
chore(statics): remove scoped concurrent access lazy_statics
chore(disk): add optional SQLite init query
perf(smart): add oneshot rewrite handling
chore(page): prep outputsink
feat(io_uring): add uring for connect_phase linux
perf(runtime): add dedicated thread for request connect
perf(page): add max limit streaming chunk size
chore(website): add connect layer concurrency limit
chore(page): add goaway retry
perf(page): add dynamic rewriter chunk size
chore(crate): remove htm5ever crate
chore(docs): fix crate features disk and balance
chore(test): fix crawl subscription
chore(static): fix concurrency access lazy_static
chore(smart): fix re-rendering wait_for
chore(disk): fix encoding url SQLite
feat(disk): add hybrid disk storing links
chore(abs): fix handling protocol
chore(chrome): add to block list
chore(website): fix http crawl
chore(chrome): add to block list
chore(abs): add safety check joining
chore(abs): fix abs handling domain
chore(page): fix block asset handling
chore(chrome): add to block list
chore(chrome): ignore block_js on top pages
chore(chrome): fix adblock compile
chore(chrome): fix allowed critical rendering script interception
chore(chrome): fix cdp serde support test
chore(chrome): fix frame inline navigation
chore(chrome): add to block list
chore(chrome): add build generate blocking
chore(chrome): add to block list
chore(docs): fix budget feature flag [#236]
chore(chrome): add to block list
chore(chrome): add to block list
chore(chrome): add to block list
chore(chrome): add to block list
chore(crate): bump ego-tree@0.10
chore(chrome): add reusable remote connect client
chore(chrome): add tokio-tungstenite
chore(balance): add simple rebalance
feat(page): add remote_addr feature flag [#226]
chore(chrome): add to block list
chore(chrome): add to block list
chore(crawler): add to block list
feat(balance): add global semaphore balancing
chore(chrome): add ad and analytics block list
chore(chrome): add ad and analytics block list
chore(crawler): add blocklist items
chore(crawler): add blocklist items
chore(chrome): fix xhr interception
chore(chrome): fix bytes timeout
chore(chrome): add block list items
chore(chrome): add block list items
chore(chrome): add linkd custom blocker
chore(chrome): revert adblock-rs default
chore(chrome): add adblock items
chore(chrome): default adblock engine
chore(chrome): fix other and manifest ignore
perf(chrome): add hydration ignore and custom website interception layer
chore(smart): add meteor.js detection
chore(chrome): add ads and tracking ignore list
feat(chrome): add page bytes collecting
perf(chrome): add xhr interception
chore(crawler): add full analytics ignore
chore(page): ignore js void
chore(chrome): add analytics ignore scripts
chore(transform): fix streaming repeated text
feat(transform): add send text transform streaming
feat(transform): add transform_content_send async streaming
chore(interning): add optional string-interning
chore(page): fix url encode handling mismatch
chore(website): fix crawl establish domain removal [#233]
chore(page): fix page link return full urls
chore(clippy): fix lint
chore(clippy): fix website display
chore(website): add url log info
chore(website): fix throttling request
chore(page): add content-type encoding respect streaming rewriter
chore(website): fix delay handling [#232]
chore(examples): fix default entry
chore(crawler): add default js enable intercept
chore(website): fix limit budget breaking
chore(crawler): fix backoff thread infinite loop
chore(tracing): fix tracing ignore
chore(backoff): add backoff max timeout
chore(chrome): add join wait
chore(chrome): add max wait for
chore(website): add semaphore cancel release
chore(chrome): add init state handling poll
chore(abs): fix page paths infinite handling
chore(tokio): remove parking_lot
chore(chrome): revert page valid url parsing
chore(chrome): add invalid network status code handling
chore(redirect): add final url handling errors
chore(logs): fix fetch error message
chore(tracing): skip attempt navigation
chore(chrome): fix browser close on initial erorr
chore(abs): add ignore protocol list
chore(abs): remove extra asset handling
perf(abs): remove extra clone on abs handling
chore(website): fix abs char boundary handling
perf(website): add conditional url parsing budget check
chore(website): fix protocol merging resources
chore(website): fix budget immediate break
chore(crawler): fix absolute url handling
chore(website): fix shutdown break loop
chore(tracing): add shutdown tracing
chore(tracing): fix establish link call
chore(tracing): add skip params
chore(tracing): add fetch tracing
chore(chrome): fix intercept passing ignore
chore(examples): add chrome remote connection
chore(tracing): add main entry trace
chore(transform): fix text transform ignore elements
chore(cache): fix build default cache backend
chore(transform): fix text ignore handler
chore(chrome): fix request interception auth enabling
chore(chrome): fix connection loop hanging
chore(tracing): add named tasks
chore(website): remove extra chrome crawl method
chore(chrome): fix semaphore drop on error
chore(trace): add tracing tasks/sets
chore(crate): add tokio tracing flag
chore(transform): fix text transform ignore
chore(page): fix full_resources [#230]
chore(website): fix streaming hang
chore(crawler): add rewriter text extracting
chore(cache): fix http mem cache backend
chore(semaphore): fix semaphore handling tasks
chore(chrome): add proxy-authorization cdp errors
chore(chrome): remove set_headers extra clone
chore(chrome): add loopback only localhost
chore(chrome): add localhost loopback remote connection
perf(transform): add rewriter markdown handling
chore(chrome): prep capturing anchor event listeners no hrefs
chore(page): add safe char handling extensions
perf(page): collect bytes without sink
perf(page): add stream full resources
perf(smart): add atomic re-render handling
perf(page): add streaming chunks rewrite
chore(chrome): add only success check handling
chore(page): add box html bytes
perf(page): pin html pre-parsing
feat(config): add concurrency limits builder method
perf(smart): add ahocorasick js handling search
perf(smart): fix js need determination
feat(page): add nextjs build ssg path handling
chore(chrome): add page url final redirect handling
perf(chrome): add skip other resources
chore(website: add full resources only_html check
chore(config): add only_html builder config
chore(config): add only_html configuration
chore(chrome): add reqeust timeout
chore(chrome): add serde device type
chore(chrome): add dismiss_dialogs config
chore(chrome): add alert polyfill
feat(real_browser): add randomize viewport
chore(website): add missing with_wait_for_idle_dom builder
feat(chrome): add web automation wait for dom
chore(page): fix page absolute join
chore(website): add increased timeout error handling
chore(crawl): fix joinset blocking task
chore(transformations): fix filter markdown/commonmark
chore(openai): add json_structure handling js browser events
chore(smart): fix establishing whitelist/blacklist
chore(budget): fix whitelist/blacklist budgeting
chore(transform): add dynamic rewriting html
chore(transforms): add optional ignore tags
chore(smart): add layui.js and jquery detection
chore(fs,chrome): fix compile viewport handling
chore(openai): fix response_format types
chore(transformations): add filter_svg
chore(encoding): fix auto encoding detection
chore(page): add network error status code defaulting
chore(website): add initial status code handling
chore(smart): fix smart page links handling
chore(page): fix link handling protocols
chore(utils): fix try exact match
chore(chrome): add automation all routes handling
chore(crawler): fix tld and subdomain handling
chore(chrome): fix window navigator stealth handling
chore(transform): revert default output transformed encoding
chore(encoding): add html lang auto parsing
chore(transformations): add html file extensions list
chore(transformations): add root selector across formats
chore(page): add waf_check exposure
chore(transformations): add filter svg markdown
chore(links): fix default robots handling check
chore(encoding): fix regex build auto_encoder
chore(transformations): fix script encoding
chore(transformations): add html_to_xml format
chore(utils): fix serde serializing selectors
chore(config): cleanup unused interception configs
feat(utils): add xpath support
chore(chrome): add bezier mouse movements
chore(config): add serializable crawl configuration
chore(page): add empty page validation
chore(backoff): add gateway backoff retrying
chore(status): add rate limit and server status tracking
chore(page): add full handling status codes retrying
chore(page): fix indexing convert abs
chore(chrome): fix smart mode wait for page
chore(page): fix retry handling
chore(chrome): add smart mode wait for navigation
chore(chrome): add return headers and cookies
chore(encoding): fix smart mode loss encoding
chore(page): fix url join empty slash
chore(chrome): patch logs stealth mode
chore(chrome): add waf mouse movement
chore(chrome): add spoofed plugins
chore(smart): add smart mode chrome retry
chore(chrome): add extra http headers
chore(chrome): fix request interception remote sockets
feat(website): add retry and rate limiting handling
chore(chrome): add cookie setting browser
chore(chrome): fix concurrent websocket page connecting
chore(chrome): fix proxy handling websockets
chore(website): add semaphore negating
release(minor): 2.6.19
chore(website): add drain_links
chore(crate): fix all feature flag compiling
chore(page): add initial start domain tracking
chore(crate): fix hickory typo
chore(crate): default tls reqwest_native_tls_native_roots
chore(config): add preserve host header configuration
chore(links): add default type
chore(spider): revert parent host change
chore(page): fix link parsing redirected domains
feat(transformations): add spider_transformation crate
perf(cpu): add bucket backend string interning
perf(mem): add buffer backend string interning
perf(website): add string interning links visited
chore(chrome): add public interception
chore(smart_mode): revert set_content
chore(docs): update intro title about
chore(utils): add trie match all base path
chore(examples): add css scraping example
perf(chrome): add major screenshot performance custom command
chore(chrome): fix chrome interception setup
perf(chrome): add concurrent request interception
chore(chrome): fix navigation setup sub pages
chore(chrome): add chrome response headers
chore(chrome): fix frame establishing
chore(encoding): add html parse auto encoding
chore(sitemap,chrome): fix compile intercept
chore(chrome): remove intercept tasks on complete
release(minor): 2.2.13
Fix infinite loop (#213)
chore(chrome): fix limit 1 handler hang
chore(chrome): fix handler hang
chore(chrome): add valid page check
chore(chrome): fix screenshot timeout
chore(chrome): add lazy chrome env connection url
chore(deps): update auto_encoder@0.1.4 binary file check fix
chore(deps): add chromiumoxide@0.7.0 and auto_encoder
chore(page): revert page links mut
perf(page): add pre-condition asset check parsing
feat(encoding): add auto detect encoding [#207]
chore(config): fix crawl budget manual set
chore(config): fix budget reset on crawl end
chore(chrome): fix remote connect dispose
chore(chrome): fix remove connect channel send handler break
feat(page): add return page links configuration
chore(chrome): fix handler shutdown on ws unexpected
chore(decentralized): fix smart mode compile
perf(chrome): remove extra browser clone
chore(chrome): fix chrome new page hang
chore(scrape): add sitemap and smart [#206]
chore(chrome): fix chrome handler break page hangs
chore(cookies): fix missing cookies flag
chore(crate): fix zstd wasm build
chore(page): add http cookies map return
chore(deps): re-export encoding_rs
Update README.md (#203)
chore(utils): release@0.2.0
chore(scrape): fix visited_links storing
chore(page): add get_chrome_page chrome page ref
chore(v2): fix scrape API
chore(minor): remove accidental typing parent_host
chore(deps): update quick-xml@0.36.0
chore(crate): add wasm entry target
feat(chrome): add trie for execution_maps and web_automation_maps
feat(chrome): add web automation scripts
feat(adblock): add network adblocking
chore(website): add website build error
chore(fs,chrome): fix chrome fs storing [#198]
chore(chrome): fix fingerprint and initial eval scripting
perf(smart): add js assets phf set
Fixing clap issues #195 (#196)
chore(openai): fix max_tokens calc
chore(chrome): add request allow channel cancelled send
chore(page): add take parsed url
chore(utils): add clean text element extracting content
feat(spider_utils): add util crate for css scraping
chore(regex): fix extra ref compact string
chore(deps): update compact_str@0.8.0
perf(cli): add crawl performant crawl streaming
chore(cli): add sync flag crawl handling
perf(cli): add subscription streaming scraping and storing
perf(robots): add fast top pass robots check
chore(chrome): fix wait_for_selector and wait_for_event
feat(chrome): add exact page execution_scripts support
chore(serde): add wait_for page_navigations defaults
chore(chrome): add serde wait_for
chore(robots): fix robots text path matching root
chore(docs): add discord chat badge
chore(bench): add debian wget install check [#191]
chore(utils): add max bytes limiting env
chore(robots): fix robot disallow respect
chore(cli): add headless and smartmode install examples
chore(website): fix sitemap regex compile
chore(minor): fix license path
chore(deps): update regex@1.10.5
feat(whitelist): add whitelist only paths
chore(budget): fix budget overflow
feat(sem): add shared queue strategy
chore(chrome): fix non chrome agents spoofing
chore(website): fix type blacklist
chore(page): add invalid url handling
chore(cli): fix params builder optional handling
chore(chrome): fix missing fingerprint initial establish
perf(crawl): add limit detection crawling
chore(chrome_intercept): fix intercept spawn handling
perf(website): fix semaphore distribution
chore(website): fix chrome stealth handling agent
chore(website): add safe semaphore handling
chore(website): fix crawl status persisting
release: 1.95.27
chore(robots): add max delay respect
chore(website): fix crawl limit immediate exit
chore(cli): fix limit respecting
chore(crate): unpin smallvec
chore(robots): fix respect robots [#184]
chore(website): add dynamic sempahore multiplier
chore(deps): bump chromiumoxide@0.6.0
feat(reqwest): add zstd support
chore(cli): fix serde_json pin
chore(openai): fix gpt_config default serde deserialization
chore(examples): fix subscribe stdout end sequence
chore(page): add set_url page
chore(openai): add tracking, ads, and meta default clean
chore(openai): remove hidden content from request
chore(openai): compress base prompt
chore(chrome): remove unnecessary page activation
chore(openai): add link and iframe to clean removal
chore(openai): add default clean on resource
chore(openai): fix smart mode passing target url
chore(openai): revert prompt spaces
feat(openai): add error tracking request
chore(openai): fix spaces base prompt
chore(html5ever): add comment endline tokenizer parse fix
feat(real_browser): add http simulation headers
feat(cache): add chrome caching between http
chore(cache): add chrome hybrid caching
feat(subscriptions): add semapre capacity matching
chore(guard): add subscription auto drop cancel
chore(openai): add fallback full context reduction
feat(cache): add top level cache manager
chore(chrome): fix blocked content waf detection
chore(chrome): add interception handling waf detection
chore(website): add missing smart mode init status handling
chore(website): fix blocked status code reset
chore(chrome): add proper status code detection
feat(chrome): add dynamic chrome connections
feat(spoof): add referrer spoofing
feat(ua_generator): add verified real user-agents ua_generator@0.4.1
chore(real_browser): add single call to handle iframe
chore(chrome): fix intercept hang
chore(chrome): fix domain host validation interception
chore(real_browser): fix frame click action
chore(real_browser): add duration timeout wait iframe
chore(real_browser): add explicit form iframe checking protected bypass
chore(docs): fix benchmarks target url
chore(website): fix url getter function name
chore(crate): unpin reqwest 0.12.4 - 0.12
chore(examples): update default domain spider.cloud
chore(docs): fix repo badges
chore(openai): add messages trim_start
chore(worker): fix headers flag compile
chore(docs): update cloud hosted link - spider.cloud
chore(chrome): fix proxy server headless connecting
feat(openai): add screenshot js execution after effects
feat(openai): add response caching
docs: fix broken glob url link (#179)
chore(openai): add detailed gpt results output
chore(parser): update fast_html5ever@0.26.4 crate compile
chore(docs): add subscribe_guard feat flag req note
chore(docs): add examples cargo doc
chore(openai): add custom api key config
chore(docs): add subscribe example
chore(docs): fix chrome missing link wait_for_delay
chore(website): add url consistency
chore(openai): fix prompt deserialization
chore(configs): add prompt deserialization
chore(configs): add option openai configs simple serde
chore(openai): add path matching prompt determination
chore(config): cleanup zero cost sub features
chore(crate): remove hard pins on common crates
Fix typo in README file (#176)
chore(website): fix sitemap chrome build
chore(crate): remove serde pin
chore(openai): fix compile chrome
feat(rss): add rss support
feat(fingerprint): add fingerprint id configuration
chore(page): return all page content regardless of status
chore(openai): fix svg removal
feat(openai): add extra data gpt curating
chore(openai): add credits used response
feat(openai): add multi driven prompts
feat(real_browser): add real browser feature flag for chrome
chore(crate): default enable http cookies
chore(page): add cold fusion file detection
chore(openai): add html_rewriting for max_tokens
chore(links): allow extra links assigning all states
chore(screenshots): fix screenshot format casing
chore(examples): fix chrome_scheenshot_with_config intro
chore(openai): add serde gpt configs
chore(docs): update clarity on openai flag env key
chore(docs): add OpenAI section
chore(openai): add explicit model checking
chore(openai): add GPTConfigs new shortcut
Openai/chrome driver (#174)
chore(tokio): enable io-std feature
chore(chrome): fix semaphore limiting scrape
chore(docs): remove discount hosted
chore(deps): update reqwest@0.11.26
feat(chrome): add evaluate on new document script handling
bench(deps): update criterion@0.5.1
chore(guard): add sub guard full crawls
chore(examples): fix queue gathering alt pages
feat(q): add mid crawl queue
chore(website): add set extra links extended crawls
chore(website): add set_url re-usablity
chore(smart): fix smart mode http defaults
feat(chrome): add auth challenge response
chore(sitemap): fix chrome sitemap handling
perf(chrome): add direct page navigation
bench(docs): fix page count
feat(chrome): add chrome_headless_new flag
chore(chrome): fix page concurrency
chore(page): fix network wait_for_event
chore(chrome): add pathbuf outdir
chore(chrome): fix page chrome close
chore(chrome): add conditional chrome close flag chrome_store_page
chore(chrome): fix wait_for_idle_network break
chore(chrome): fix intercept early exit
chore(chrome): fix page hang
chore(clippy): fix join_handle warnings
feat(chrome): add wait_for configuration delay, selector, and idle network
chore(chrome): add prep for custom wait_for_events
chore(chrome): fix final_url page when wait_for_network_idle
chore(chrome): fix page url determination
bench(perf): add local dev server for testing (#167)
chore(chrome): fix wait_for_network page extracting
chore(chrome): fix bytes handling
chore(website): remove extra conditional builder methods
chore(website): add direct assign http client method
chore(docs): add spider hosted coupon code
chore(docs): fix warnings across feature flags
chore(deps): update tokio@1.36.0
release(minor): add response headers feature
Felix engl/main (#164)
chore(cli): add accept_invalid_certs arg
chore(page): fix subdomain detection
feat(config): add with_danger_accept_invalid_certs builder method
chore(website): add url empty whitespace handling
chore(cli): fix full resource include external domains
feat(cli): add crawl page limit
chore(page): fix external domains include
chore(page): add shtml resource allow list
perf(semaphore): fix dynamic sem limit scaling
perf(runtime): remove yielding switching
feat(chrome): add new waitforidlenetwork timeout control
chore(sitemap): fix gathering all links
feat(chrome): add wait_for_network_idle
chore(lib): fix clippy suggestions
chore(examples): add subscribe_download example
chore(examples): fix download file path
chore(sitemap): fix extracting multiple sitemap urls
chore(website): fix un-controlled shutdown
chore(website): fix http scrape persist links between crawls
chore(website): re-use crawl handle check
chore(docs): add subscribe_guard example
chore(chrome): fix chrome initial scrape page set
chore(chrome): fix chrome glob compile
chore(scrape): fix store initial page
perf(scrape): add shared arc
chore(website): add error log scrape
chore(sync): add subscription guard
chore(chrome): add screenshot default compilation
chore(sync): add unsubscribe channels
chore(chrome): fix chrome screenshot storing
chore(chrome): return bytes screenshot helper
feat(chrome): add viewport configuration
chore(docs): cleanup website & config
feat(config): add full_resources manual set
chore(config): fix cache compile chrome
perf(page): fix inline hot link extracting
chore(chrome): fix cache enabling/disabling
test(cache): fix enable cache
chore(config): fix limit configuration builder
chore(config): add simple crawl limit builder method
feat(chrome): add override locale emulation
feat(chrome): add timezone override config
chore(chrome): fix request interception execute self hosted scripts
chore(examples): add glob example
feat(config): add re-usable configuration building
perf(cron): add async_job@0.1.4 lock fix
chore(deps): update async_job@0.1.3 panic removal
chore(sitemap): add persistable chain crawls
feat(sitemap): add crawl_sitemap
perf(encoding): add html size buffer allocation streaming
feat(encoding): add dynamic streaming encoding html
chore(configuration): add full builder confs
chore(sitemap): fix scrape raw handling
chore(website): add full builder defaults
perf(budget): remove extra string allocation validation
perf(robots): remove extra mem allocation link validation
perf(control): add rwlock crawl control
chore(stealth): add default method
feat(redirects): add redirect limit expose with server respects
chore(control): add crawl_id for explicit website control
feat(depth): add crawl depth level control
chore(chrome_intercept): fix intercept redirect initial domain
feat(cache): add caching backend feat flag (#156)
chore(chrome_intercept): fix host match blocking assets
chore(redirects): fix subdomain checking domain
feat(chrome_intercept): add chrome_intercept request interception flag
feat(redirect): add transparent top redirect handling
chore(decentralize): fix glob build
perf(scrape): remove extra clone page broadcast
feat(chrome_store_page): add feat flag chrome_store_page and screenshot helper
chore(smart): fix nextjs static app detection
perf(smart): add re-render html dispose
chore(website): add smart mode method default
perf(smart): prevent static app re-rendering
Smart mode (#154)
chore(website): add with_cookies builder method
chore(website): add auto handle transport
chore(bench): fix criterion table count
chore(website): revert raw method flags
chore(control): fix control task abort after crawl
feat(chrome): add chrome_screenshot feature flag
chore(website): add website.stop handling shutdown
chore(crate): re-export async_job
chore(website): add re-crawling clearing
chore(cron): move cron feature to async_job crate
chore(exports): add smallvec re-export
chore(website): add chrome flag over raw HTTP methods
chore(website): add enum string crawlstatus crontype
chore(docs): fix fs feature default
chore(docs): add node.js port docs
chore(website): add crawl invalid state
feat(client): add experimental http3 support
chore(cron): add ref runner stop
chore(feat): add public features
chore(napi): add rust_tls flag
feat(napi): add napi support
chore(cron): remove default feature
feat(cron): add cron feature flag (#153)
feat(cookies): add cookie jar optional feature
chore(website): remove public expose setup
chore(chrome): fix raw page build
chore(deps): bump tokio@1.34.0
chore(ahash): bump ahash@0.8.6 AES ARM instructions fix
chore(website): add crawl_concurrent_raw method (#152)
chore(chrome): add main chromiumoxide crate
release(minor): 1.49.5
Return status code on error (#151)
test(website): fix invalid configuration test
chore(website): fix crawl re-run host determination
chore(cli): add error output incorrect website url
chore(cli): add http check cli website url
chore(chrome): fix chrome headless headful args
chore(docs): cleanup website
chore(feat): add reqwest_json feature flag re-export use
chore(scraper): fix text processing ignore script/style
feat(page): add status code and error message page response (#148)
chore(page): fix subdomain entry point handling root (#146)
chore(docs): add missing sync default
chore(config): add missing with_on_link_find_callback [#145]
chore(chrome): add error log instance launch
chore(docs): spider cloud hosted intro
chore(chrome): update chromiumoxide_fork@0.5.12 safe ws connection handling
chore(chrome): fix tungstenite remote headers ddos vulnerability
feat(external): add wildcard handling all domains found
chore(docs): remove early discount
feat(cli): add external domains grouping
feat(cli): add crawl budgeting
chore(docs): fix spider hosted promo
chore(docs): add hosted early launch oss discount
chore(scraper): add element attribute extracting
docs(fix): fix error in "scrape/gather html" section (#137)
chore(deps): remove matches
bench(node): disable bench invalid output
chore(sync): fix build missing broadcast
chore(docs): add respect_robots_txt delay note
ci(bench): add full scopes
chore(docs): add budget feature flag req
chore(worker): fix scrape fetching content
chore(redirect): add fast chrome redirect determination
chore(chrome): fix build page missing mut
chore(budget): allow budget public access
perf(budget): use flag to prevent degradation
feat(budget): add crawl budgeting pages
feat(page): add page redirect destination exposure [#127]
chore(worker): fix description of the crate
chore(glob): fix link callback #136
chore(docs): add hosted pay as you go example display
chore(cli): fix links json output
feat(external): add external domains grouping
chore(benches): default proxy to spider worker
chore(deps): update bytes@1.5.0
chore(sitemap): fix sender drop
feat(sitemap): add sitemap crawling feature flag
chore(chrome): fix ip connecting
chore(chrome): fix remote page closing
chore(chrome): fix proxy remote interface connections
chore(chrome): fix remote browser droping
chore(chrome): fix remote debug ws find
feat(chrome): add CHROME_URL env var remote connections
chore(crawler): fix chrome_cpu flag launch args
chore(chrome): add CHROME_BIN env variable handling
feat(chrome): add stealth mode
feat(chrome): add proxy networking
feat(chrome): add chrome_cpu feature flag
chore(chrome): fix content extracting after navigate
chore(fetch): remove duplicate funcs
chore(api): move chrome to features
chore(crawl): remove crawl sync api
chore(chrome): fix chrome browser shutdown
feat(chrome): enable chrome rendering page content
feat(website): add crawl status
chore(docs): add missing feature flag info
chore(exports): add tokio_stream re-export
feat(page): add duration tracking
chore(docs): add new hosted service link
chore(website): add domain parsed getter
chore(website): add domain name getter
chore(page): fix trailing slash url getter
chore(full_resources): fix capturing link tag
chore(cli): fix resource extensions [130]
chore(pages): fix full_resource gathering scripts
feat(page): add byte storing resource (#131)
chore(cli): add regex and full_resources flags
chore(sync): fix initial link sub capture
chore(sync): fix subscribe handling
feat(sync): add broadcast channel subscriptions v1.36.0
chore(utils): fix cfg control feature
chore(lib): add full re-exports
chore(scrape): add resource in on_link_find_callback
chore(page): fix domain name clean handling
perf(url): update percent-encoding@2.3.0
perf(css): improve link tree parsing nodes (#126)
chore(website): fix crawl establish link handling
chore(website): fix cfg crawl establish
chore(bench): fix env var massive bench
feat(headers): add http header configuration
perf(regex): use regex set one pass (#125)
chore(deps): pin versions and update tokio@1.29.1
chore(docs): add builder pattern configuration usage
feat(builder): add builder pattern [#115]
chore(spider): add .jsp file detection support
chore(bench): add env crawl massive target url
chore(feats): remove spoof ua_generator default
chore(regex): fix compilation
feat(cli): add static markup downloading
chore(deps): update jsdom@0.0.11-alpha.1 SO fix
chore(deps): update deps to latest
chore(docs): update bench speed compare
perf(workers): add increased semaphore limits (#123)
chore(decentralized): revert semaphore
chore(page): fix extra url parse selector gathering
chore(page): remove double process conditional check
chore(page): fix asset detection
release(minor): add feature js script link extracting v1.31.0
feat(js): add init of script parsing (#121)
chore(page): fix crawl processing ignore assets
chore(docs): fix crawl sequential docs
chore(worker): fix cfg scrape build
perf(worker): drop semaphore usage decentralized
feat(worker): add tls support
test(glob): fix http proxy test case
chore(robots): fix robots parser link gathering
chore(request): add custom domain redirect policy
chore(glob): fix glob crawl establish
chore(crawl): fix crawl asset detection and trailing start
perf(strings): use case_insensitive_string as primary string (#117)
chore(page): fix trailing slash links find
chore(decentralized): fix transport type response return
bench(decentralized): add decentralized worker split case (#116)
release(minor): feat url glob and hybrid temp disk support
chore(docs): add glob section
feat(fs): add temp storage resource handling (#112)
chore(glob): fix base subdomain tld crawls
URL globbing (#113)
chore(network): fix status code handling valid resource
chore(proxy): fix proxy interception and handling
chore(socks5): fix proxy handling
ci(proxy): add local socks5 server
test(spider): fix crawl proxy case
release(spider): 1.28.0 proxy support
feat(proxy): add request proxy support
perf(io): add bytes_stream body handling
chore(docs): fix benchmark page count
perf(sem): fix semaphore count (#109)
chore(crawl): fix link domain handling
chore(blacklist): fix option handling
feat(config): add http2_prior_knowledge support for http2 request
perf(regex): compile regex start of crawl
chore(worker): fix selector handling
chore(docs): fix feature headings
chore(docs): fix website user_agent assign
feat(flexbuffers): add flexbuffers serde decentralization (#105)
perf(crawl): remove unused selectors building
feat(full_resources): add flag for gathering all content related to domain
feat(serde): add serde de/serialize case-insenstive strings
chore(website): fix getting pages
chore(worker): add string_concat url format
perf(decentralized): add link gathering decentralization (#103)
feat(control): move crawl control under feature flag
chore(decentralized): fix proxy traffic handling
feat(decentralize): add start of decentralized worker
chore(html5ever): add create_comment stub
chore(crawl): handle invalid domain panic
chore(minor): build client without checking
perf(crawl): remove extra yield on task join
perf(async): add current handle runtime set
chore(examples): fix htr react example
perf(crawl): fix page link stream switching logic
perf(scraper): add html5ever fork ignore comments + errors
perf(task): fix semaphore task limiting (#102)
perf(crawl): add single shared arc concurrent
chore(page): add optional html
bench(parse): add stream scraper bench marks
chore(crawl): fix scheme mismatch crawling
perf(crawl): reverse logic for extension handling
bench(crawl): add massive crawl benchmark init
chore(crawl): fix fragment handling of url ego walk
chore(crawl): fix extension handling of domains
chore(crawl): fix safe ignoring extra dot ext
perf(parser): add fast concurrency setup (#100)
chore(crawl): fix crawl reset active links
perf(scraper): add fast element node creation
perf(scraper): fix cache and hasher
build(deps): add scraper to packages
perf(crawl): reduce memory usage gathering links
perf(crawl): reduce memory usage and make cb optional
perf(css): update css3 selectors fast_scraper@0.14.3
perf(crawl): reduce mem allocations css selectors and network client
perf(ua): fix ua memory alloc for client
chore(crawl): fix link callback handling on pre gather
chore(robots): fix sitemap last write time
perf(crawl): add join handle task management (#99)
perf(scrape): add compact selectors
perf(scraper): add ahash imp of scraper fork
perf(crawl): add memory efficient strings (#98)
feat(accuracy): improve crawl handling very large websites
chore(crawl): add optional duration to disable timeout
chore(docs): add reference examples instead of slow page gathering
chore(docs): add missing scrape section
build(tokio): remove unused net feature
perf(client): add mass scan configs (#97)
perf(crawl): add pre link gathering boost
perf(crawl): remove mem usage on robot parser
perf(crawl): add optional heap blacklist_url
chore(crawl): remove double hashmap concurrent runs
chore(crawl): add heap links_visited usage (#95)
chore(crawl): remove active crawl link storing in self
chore(crawl): fix loop break
perf(tokio): enable parking_lot (#93)
chore(crawl): add end breaking
chore(minor): remove unnecessary await on handler config
perf(inlining): fix cold path inline removal
benches(cargo): fix install deps
chore(hash): fix borrow handling links visited (#91)
perf(inline): add inline opts (#90)
perf(links): add ascii case-insensitive handling (#89)
perf(parsing): add fast link serialization extracting
chore(docs): fix crawl doc
chore(scrape): add optional page setup
chore(docs): fix robot parser example
perf(crawl): remove static url handling robotparser
perf(crawl): add top level subdomain tld base extracting
chore(crawl): add case insensitive and trap handling
Converted clippy warned single character strings to chars (#87)
chore(crawl): add top throttle setup
chore(crawl): fix link delay between
perf(crawl): add top level selector building
chore(bench): add hyperfine mac benchmarks
chore(ignore): add img and xlsx to ignore
Compiler Warnings (#82)
chore(parser): add .mov file ignore

What's Changed

feat: add on_link_blocked_callback for robots.txt blocked URLs by @zanmato in #394
fix(website): crawl_raw uses raw sitemap chain on chrome builds by @zanmato in #393
Fix 500ms timeout for proxy error statuses 598/599 by @waamm in #395
• Fix Unbounded Task Spawning in Sitemap Crawl by @Uinelj in #397
chore(deps): bump actions/checkout from 4 to 7 by @dependabot[bot] in #399
browser: per-crawl control map for optional chrome enhancements by @j-mendez in #400
fix: concurrency/memory/panic audit hardening (v2.52.7) by @j-mendez in #401

New Contributors

@zanmato made their first contribution in #394
@waamm made their first contribution in #395
@Uinelj made their first contribution in #397
@dependabot[bot] made their first contribution in #399

Full Changelog: v2.48.4...v2.52.2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

v2.52.2 — CLI authenticate for Spider Cloud

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

Changes since vv1.7.5

What's Changed

New Contributors

Contributors

Uh oh!