Repository navigation
Releases
Release v0.2.0
First stable release on crates.io. All in-house dependencies unified to stable versions (minor precision), alchemize switched to the crates.io registry, no path/git dependencies remain in this repository, and the cargo publish pipeline is unblocked. This section incorporates the former [Unreleased] section.
Features
- i18n system rewrite: the messaging layer switched to the
sdforge::i18nhost registry,locales/*.ftlfiles are embedded at compile time (src/i18n/catalog.rs,include_str!+ line parsing at startup viaregister_translation), and deployment artifacts no longer ship an on-disklocales/directory; the key guard was upgraded to bidirectional consistency between the embedded manifest and the on-disk directory (tests/unit/i18n);t_with_args/tr_log_argsparameters simplified to&[(&str, String)]({ $name }replaced by name) - crates.io release pipeline: with all dependencies on the registry,
cargo publishis unblocked;release.ymlmigrated to the final family state (OIDC trusted publishing preferred +CARGO_REGISTRY_TOKENfallback, idempotent creation via agh releasescript step) - Trial integration of the alchemize html engine (path dependency, converter=alchemize)
Bug Fixes
- rag-local dead on arrival at runtime: the vecboost dependency with
default-features=falsestripped out theonnxmain engine, while the provider was pinned toEngineType::Candle— the vecboost factory fails fast on an engine that was not compiled, sorag.enabled=true+embedding_provider="local"failed at startup. Fix: setfeatures=["onnx"]explicitly on the dependency and pin the provider toEngineType::Onnx - Two
needless_boolcases under nightly clippy: the sitemap seed guard now returns the feed result directly, and the cache_mode POST check uses a negated match expression - SDK integration test E0061:
I18nBundle::loadinsdk_api_testchanged to a two-argument call (a stale test missed during the switch to sdforge host registration) - acceptance teardown
single_match: logger shutdown intests/acceptance/teardown.rschanged toif let(the release workflow'sclippy --testswould flag it) - BDD infrastructure: closed out two accompanying defects — swallowed warn logs and a Ryuk container leak
- Closed out two release-gate test defects — acceptance teardown clippy single_match and the Python create-scrape assertion of 201
- sdk_api_test's I18nBundle::load changed to a two-argument call — a stale test missed during the switch to sdforge host registration (root cause of CI integration leg E0061)
- fmt gate changed to target two packages with -p (--all affected inactive path dependencies)
- coverage job narrowed to --lib (90.12%; a cold build of all targets timed out)
- Coverage timeout increased 45→75 minutes (measured 44m25s for a cold compile after dependency changes; cancelled jobs do not write back the cache)
- deny switched to run directly on the runner + CodeQL added sibling checkout (the Docker action cannot see symlinks)
- Added vecboost to the sibling checkout chain (a path dependency of alchemize)
- All cargo jobs check out the alchemize sibling (needed during path dependency resolution)
- Integration Tests timeout increased 25→40 minutes (measured 25m15s, a borderline timeout that was cancelled)
- Coverage job timeout increased 30→45 minutes (measured 30m16s, a borderline timeout that was cancelled)
- Closed out two accompanying infrastructure defects — swallowed warn logs and a Ryuk container leak
- Adjudication reason enum aligned with API literal values (including spaces)
- CodeQL false-positive adjudication switched to PATCH (the current REST endpoint method)
Other Changes
- Markdown converter default switched to alchemize: the default of
[markdown].converterchanged frommolitoalchemize(in-house html engine: htmd base chain + in-house deep chain, with built-in empty-output safeguards, honest degradation, and math$TeX$fidelity routing); thecontentfeature absorbs alchemize, andhtmdis retained as a pass-through diagnostic arm. Equivalence measured: 13 cases (5 real pages + 8 synthetic), 10/13 byte-identical (only trailing newlines differ); math pages improved from leaking MathML to$TeX$; 2 navigation-element heuristic differences (no content loss); throughput on wiki 1.06MB×20: moli 105.6 vs alchemize 96.8 MB/s (-8%); assembled-chain output byte-for-byte identical to alchemize native on 5 real pages - MSRV 1.97 → 1.97.1: aligned with alchemize's
rust-version; CI toolchain matrix and README/FAQ/CONTRIBUTING text synchronized - Coverage job narrowed to
--lib: after alchemize entered the tree, cold builds of all targets repeatedly exceeded 75m (cancelled jobs do not write back the cache); lib-only measured 90.12% (integration tests added <0.01pp), with the 80% gate unchanged - CI gate criteria: fmt changed to target two packages with
-p crawlrs -p crawlrs-examples; coverage/integration/BDD job timeouts raised per measurements (30→45→75 / 25→40 / 30 minutes) - All in-house dependencies unified to stable versions (minor precision): confers
0.6, dbnexus0.6.1, garrison0.9, inklog0.3, limiteron0.3, nebulaid0.3, oxcache0.5, sdforge0.5, trait-kit0.5, vecboost0.3, fluxweir0.1(resolved 0.1.2), alchemize0.1; rc pins on direct dependencies cleared to zero - alchemize switched to the crates.io registry: removed
path = "../alchemize", socargo publish/cargo packageare no longer rejected due to a path dependency; the CI.github/actions/alchemize-siblingcheckout chain was removed (a clean checkout resolves directly) - fluxweir returns to the registry: from 0.1.1 it provides
with_pool/build_app_with_pool/TrustedIdentity, so the rev-pinned git dependency was removed and deny'sallow-gitcleared - moli-html2md dependency and conversion chain retired: deleted
moli_adapter.rs/moli_converter.rs/converter_chain.rs(the moli-primary + htmd-fallback chain), removeddep:moli-html2mdanddep:ego-treefrom thecontentfeature; thecrawlrs_md_converter_fallback_totalmetric was removed along with the chain (alchemize's built-in fallback is no longer counted separately) - fluent-bundle dependency retired: zero references after the messaging layer unified on
sdforge::i18nhost registration; theI18nSettings.locales_dirfield was removed as well (theCRAWLRS__I18N__LOCALES_DIRenvironment variable is no longer read) - Migration note:
CRAWLRS__MARKDOWN__CONVERTER=moli(the old default) is rejected during configuration validation (startup fails); it must be changed toalchemize(the default) orhtmd - security workflow permissions: added
checks: write— rustsec/audit-check needs it to write back the check-run (R-17: Resource not accessible by integration) - CodeQL alert adjudication workflow: made false-positive adjudication idempotent (skips already-adjudicated alerts), aligned the adjudication reason enum with API literal values, and switched to PATCH (the current REST endpoint method)
- Integration Tests timeout 40→70 minutes — a ~45m cold compile after a dependency lockfile change caused job 7e64a2b to be cancelled
- RUSTSEC-2025-0141 bincode unmaintained: same-source exemption in two files
- 0.2.0 stable release wording — new CHANGELOG section, README/architecture/API/guide versions and dependency baseline aligned
- In-house dependencies unified to stable versions + version bumped to 0.2.0 — minor precision, alchemize switched to the crates.io registry
- release.yml migrated to the final family state — OIDC trusted publishing + token fallback, gh release script step; removed the alchemize sibling checkout
- Cleared two nightly clippy needless_bool cases — the sitemap seed guard returns the feed result directly, and the cache_mode POST check uses a negated match expression
- Added checks: write — rustsec/audit-check needs this permission to write back the check-run (R-17: Resource not accessible by integration)
- Unified interfaces across the crate family — trait-kit rc.14 (PluralCategory re-export) + sdforge rc.11 + confers rc.13 (as involved in this repository), with no breaking changes to the public API
- Removed the dead fluent-bundle dependency (the messaging layer already unified on sdforge::i18n host registration, zero references)
- Rewrote the in-house Fluent i18n system + pinned the vecboost onnx engine + batch of engine/worker fixes
- pre-commit cargo-fmt hook synced to the -p criteria, closed out remaining documentation items
- Removed the vecboost sibling checkout (audio cannot be enabled; verified that resolution succeeds without it)
- bump actions/upload-artifact from 4.6.2 to 7.0.2
- bump actions/download-artifact from 5.0.0 to 8.0.2
- bump zizmorcore/zizmor-action from 0.7.1 to 0.8.1
- Test statistics re-measured after the moli chain removal (6,548 / 229,992 lines)
- Removed moli-html2md; the alchemize html engine becomes the default converter
- Synced with upstream feature shape (one feature per format; html includes the htmd base chain + deep chain)
- fluxweir switched to crates.io 0.1.1 (deny allow-git removed that source as well)
- Made the false-positive adjudication workflow idempotent (skips already-adjudicated alerts) and removed temporary diagnostics
- Added 404 diagnostic output to the CodeQL false-positive adjudication workflow
- Added a CodeQL false-positive adjudication workflow (identifier-logging FPs, with security-events permission delegated)
Related PRs / Issues
Release v0.2.0-rc.4
This release fixes the depth counter clamped at a lower bound of 0 and intermittent BDD connection closed issues; acceptance is pinned to offline mode in release tests for consistency with CI.
Bug Fixes
- CI fixes: six no-default warnings across the five Feature Matrix combinations (
RetryReasoncfg gating, unused parameters in race/sequential, and threeurl_redactioncases attributed to features via allow); the engine-moli memory-gating test's cross-submodule access changed tostate_for_test()(also fixing the_available_mibwarning); the BDD job added the PostgreSQL service and migrations steps (previously TEST_DATABASE_URL pointed to a nonexistent service, so bootstrap failed wholesale with "Connection closed") - Queue metric fix:
PostgresTaskQueuedepth counter clamped at a lower bound of 0 — crawl subtasks are enqueued via a directrepository.createwrite while workers decrement viaqueue.dequeue, so negative drift caused u64 wraparound: in debug builds a subsequent enqueue overflowed and panicked (the root cause of intermittent BDD "connection closed before message completed"), while release builds silently emitted garbagecrawlrs_queue_depth; includes a regression unit test - Security channel: added same-source exemptions in
.cargo/audit.tomlanddeny.toml(including RUSTSEC-2026-0459, where sqlx 0.9's<0.38constraint blocks the fixed version 0.39); adjudicated all 18 open CodeQL alerts down to zero (9 identifier/masked-log FPs, 2 high-entropy key-digest FPs, 2 per-request TLS option won't-fix, 5 test-fixture used-in-tests) - Comments and documentation: cleaned up task-number/review-stamp comments (in CI and worker/engine/handler comments); fixed 3 doctest errors (
record_block()/intercepted_count()/HtmdMarkdownServicepaths); aligned API_REFERENCE/FAQ/USER_GUIDE versions and health examples; re-measured README statistics (230,418 lines) - Examples: 18 examples previously always failed because
EngineClient::new()registered zero engines; addedcrawlrs_examples::default_engine_client()(ReqwestEngine) and wired it into 17 examples (mllm_scraperequires full LLM DI, so it deliberately keeps an explicit zero-engine setup to avoid ordinary HTTP success masquerading as MLLM success);basic_scrapemeasured HTTP 200 on example.com; all examples compile with-D warnings - Dependencies: html-escape 0.3.1, toml 1.2.1, uuid 1.29.2 (cargo-outdated shows only testcontainers-chain constrained items, which remain pinned)
- Verification: lib 6379/0, integration 170/170, main 790 passed/16 ignored, sdk 8/8, doctest 10/10, line coverage 90.13% (pipeline Step 9 target ≥90 met; CI threshold remains 80%), E2E matrix/static/unit/integration/bench/liveapi six stages PASS (report aggregation SKIPPED by design), BDD acceptance run 5×59/59 consecutively, clippy(default/full/agent-lib)+doc+deny+audit all green
- Clamped the depth counter at a lower bound of 0, fixing the root cause of intermittent BDD connection closed
- Pinned acceptance to offline mode in release tests, consistent with CI
- When git dependencies are present, GitHub Release uses the tag source archive and skips crates verification
- git dependency detection changed to word-boundary matching with an inline-table form
- When git dependencies block crates.io, skip explicitly and allow GitHub Release
Other Changes
- Release 0.2.0-rc.4
Release v0.2.0-rc.3
This release adds a Markdown converter chain — moli-html2md primary + htmd fallback — and a WaitFor response-level wait signal (absorbing moli --wait-response…); it fixes and closes out three-dimensional review MEDIUM/LOW items and blacklist/terminal-state invariants, closes out terminal-state reconciliation, the blacklist execution chain, and SDK crawl creation, and adds the E2E matrix, keeps temp live-test artifact directories out of version control, and moves FEATURE_MATRIX out of the repository.
Features
- Markdown converter chain — moli-html2md primary + htmd fallback
- WaitFor response-level wait signal (absorbing moli --wait-response-* semantics)
- MoliEngine — a JS rendering engine that controls an external moli serve via CDP
- Memory-watermark scheduler and js_degraded degradation flag (engine-moli feature)
- fluxweir gateway integration — unified exit for LLM/embedding/reranking with single-pool sharing
- Completed key registration + closed out detection/fallback semantics
- Made repository error classification explicit (rate limiting / optimistic lock conflicts)
- Performance baseline regression gate and status alignment — bench.yml/bench_gate.py, PERFORMANCE/CHANGELOG/README updates (T013-T017)
- Redis shared cache — [cache] backend configuration, fail-closed validation, and memory/redis dual-backend switching (T010-T012)
- GET /v1/tasks/{id}/ws real-time task status push (T006-T009)
- Supply migration directories at runtime by dialect and wire up auto_migrate (T004/T005)
- Three-dialect schema supply — separate migrations/{postgres,sqlite,mysql} directories + consistency tests
- merge: feat/searxng-meta-search — SearXNG meta-search integration
- SearXNG meta-search integration — public instance lifecycle and failover
- Hand-written provider stack retired — LLMServiceTrait established as the injection boundary
- Embedding unification baseline aligned with the rc.9 line + embed-probe gate locked in
- Ledger fully cleared — single error-response shape / full fingerprint coverage / result invariants / backend verification
- Playwright stealth fingerprint consistency upgrade — Sec-CH-UA / dual word lists / New Headless
- IdModule algorithm validation and auth-off operational logs now emitted via tr_log
- ID module configuration validation and auth-off degradation logs wired into Fluent
- Engine health monitoring absorbs garrison HealthCheck/Registry (auth-gated)
- status_code 17-arm mapping moved down into the sdforge unified table
- 42 user-visible strings/logs wired into Fluent (en default + zh key alignment)
- Vision calls support custom OpenAI-compatible endpoints + live-test target corrected (F12)
- AIMD concurrency target team gauge and increment/decrement debug logs
- Character-level provenance alignment for extraction results + MLLM hybrid AX-tree snapshots
- Extraction degradation-ladder quality gate + LLM token-window chunked parallel merge
- Playwright XHR/Fetch background response capture
- sitemap streaming as a first-class request source + soft-404 fingerprint + seen prewarm delta
- Failure-scene snapshots (error signature → HTML + screenshot) + AIMD memory-pressure linkage
- Session identity triplet binding and retirement of error_score scoring
- AutoThrottle delay convergence + 429 exponential backoff + Crawl-delay dual clocks + stall detection
- exa/tavily conditional registration + baidu unwrapping + incremental-crawl conditional requests
- stealth injection adds webgl_overrider (WebGL vendor/renderer fingerprint override + canvas read noise)
- Browser-pool per-task BrowserContext isolation (context_isolation enabled by default, with an escape hatch on the TabPool reuse path)
- per-host discipline component (polite rate limiting + 429 cooldown + bare 403 streak → TLS upgrade) + 429 Retry-After parsing wired up
- robots.txt fetch failure fail-closed fallback (short-TTL failure-flag cache) + respect_robots request-level override semantics locked in
Bug Fixes
- Closed out three-dimensional review MEDIUM/LOW items and blacklist/terminal-state invariants
- Closed out terminal-state reconciliation, the blacklist execution chain, and SDK crawl creation
- Fixed crawl terminal-state counting/matrix construction and opened up all E2E stages
- MEDIUM/LOW closeout and MySQL migration CI execution verification
- Added MySQL 010 legacy url_hash upgrade and cleared the test baseline to zero
- Fixed three-party review HIGH/MEDIUM issues and closed out remaining problems
- Completed review MEDIUM/LOW remediation (security/IP/migration/performance)
- Fixed 12 E2E defects and completed three-party review remediation
- fix(security)+refactor(core): closed out spec convergence-batch-2
- coalesce keys include the full request identity and tenant, closing content cross-talk
- Cleared 30 outstanding OCR review items (concurrency/credentials/CI guards)
- bytes version notation aligned to the minor level (1 → 1.12)
- Fixed three false-green gates and closed out UA architecture and ws quota defects
- Assertions keyed by compilation target and locale, with hardcoding removed
- Promotion probe concurrency cap + map endpoint status-code criteria aligned
- Added an in-flight gate to number-pool replenishment, eliminating concurrent duplicate replenishment
- fetch_update changed to try_update (deprecated in rustc 1.99)
- Restored the mtime redaction fix accidentally reverted by 60743dd
- Fixed zizmor's nonexistent v0.12.0 and shallow-clone gitleaks no-op, and removed the always-red cargo vet
- Wired connect_timeout throughout the production client to prevent connections from exhausting the total timeout
- js-upgrade reroute dead-end degradation — return the raw response when no JS-capable engine is available
- Ledger cleared to zero — unified 429 wrapping / error propagation / result invariants / credential hygiene
- Outstanding items converged — WKWebView family data truth / locale / arch + test gates completed
- id algorithm parsing assertions aligned with NebulaId rc.3 alias reversal
- Documentation audit defects #1-#4 — migrate subcommand, cancel route, search config consumption, scrape option exposure
- Restored auth-off default-deny semantics and the web-axum equivalent build surface (documentation audit defect #6)
- Chunking/provenance wired into the live extraction path + direct-call max_tokens cap and configurable timeout
- Added the content_extractor field to WorkerManagerDeps test construction
- Fixed stage-failure PIPESTATUS determination + search fault tolerance allows explicit validation errors through
- Fixed construction adaptations, outdated assertions, and clippy violations exposed by live gates
- Unified garrison global-singleton cross-test mutex + RBAC missing-table loudly skip
- content_extractor quality-gate Facade injected into the worker via DI
- coalesce waiters copy the leader's result before completing
- Strengthened Baidu verification-page title detection + search unit tests assemble real engines
- 429 determination short-circuits before data-JSON
- Independent profile directory per chromium instance + fallback reclamation on pool destruction
- Request-level cookie jar preserves Set-Cookie across the redirect chain
- Direct HTTP replaces EngineClient, so local LLM endpoints are no longer blocked by crawl SSRF
- Wired up the CRAWLRS_DISABLE_SSRF_PROTECTION bypass
- Removed the dead jsonwebtoken dependency — zero references across src
- single-flight waiters in the same batch share the leader's result, eliminating serial amplification of failures
- HostDiscipline lazy sweeping + toDataURL does not materialize the context
- gate changed to a per-host reservation schedule, with ≥ min_interval spacing between concurrent requests to the same host within a batch
- Rolled back tab-return browser-level cookie clearing + added host anti-bot memory regression tests
- URL normalization strips known tracking parameters + robots matching carries the query
- Absolute-ize relative links in body HTML + escape line starts in the PlainText degradation branch
- host anti-bot memory + DomStable degradation + DFS depth key + race escalation reassignment + _abck recovery
- r2 aggregation chain hardened — per-engine timeout / locale propagation / cross-engine dedup / outbound rate limiting / challenge-determination fixes
- bootstrap injects a wreq response-body limit + docs synced with audit fixes + typos allowlist for _abck
- only_main_content Markdown preserves body structure (links/headings)
- SmartSearchEngine wired to the production API + RRF dimensional normalization + Bing challenge-page detection and link unwrapping
- Bloom false positives causing permanent missed crawls + startup prewarming + automatic capacity scaling + DFS semantics corrections
- stealth CDP injection + true networkidle + wait_for wiring + pool proxy validation + tab cookie clearing
- race mode wired into anti-bot detection + Akamai detection flag corrections
- All-scheme proxy interception (Proxy::all) + wreq TLS impersonation landed + UA pool Chromium filtering
Other Changes
- Release 0.2.0-rc.3
- E2E matrix and temp live-test artifact directories kept out of version control
- FEATURE_MATRIX moved out of the repository
- Disabled the memory-watermark scheduler to eliminate machine-dependent intermittent stalls
- Feature combination matrix and contract errata (FEATURE_MATRIX)
- Exhaustive scenarios implemented — cancellation/boundary/validation matrix/batch cancellation and webhook cross-talk fixes
- Sixth wave of the complexity long tail (clear-to-zero round)
- Wave five closeout — extraction scheduling unification and billing settlement strategy
- refactor(core)+fix(security): pending items cleared to zero and complexity long-tail batch T004/T005
- Wave one convergence — refactoring of 43 high-complexity functions + three-dimensional review fixes
- Cleared 9 outstanding three-dimensional review items (security/performance/architecture)
- Behavior-preserving refactor of high-complexity paths and split route_se...
Release v0.2.0-rc.2
This release upgrades the github/codeql-action/init dependency, adds unify-rust-i18n crawlrs residual Chinese cleared to zero (T023) and unify-rust-i18n [T012] crawlrs i18n remediation, fixes the sticky TOCTOU race-condition test window 1ms→50ms… and Merge remote-tracking branch 'origin/mai…, and adds the 0.2.0-rc.2 section (Version Consistency gate) and unify codeql-action to 4.38.0….
Features
- unify-rust-i18n crawlrs residual Chinese cleared to zero (T023)
- unify-rust-i18n [T012] crawlrs i18n remediation
- RAG retrieval-augmentation/rerank/embedding providers (incorporated from a parallel workflow) + example and config updates
- Operational hardcoded values folded into configuration (max_pages/response-body limit/robots TTL + retries)
- /v1/scrape optionally respects robots.txt + configurable User-Agent
- S-tier in-house capability absorption, second batch (dbnexus/garrison)
- Absorbed in-house library capabilities (quick-win batch + medium batch)
- migrate wreq 5.3 → 6.0.0-rc.31
- upgrade garrison 0.8 → 0.9.0-rc.1 + bump self-deps to rc.2
- POST /v1/map site URL discovery endpoint (including making the SSRF switch effective)
- sitemap parsing and /v1/map business logic
- /v1/map request DTO and validation
- Wired up retention batching/timeouts/mutex (cleaned up leftover imports)
- RetentionWorker multi-instance mutex
- RetentionWorker per-category timeouts
- audit_logs batched bounded cleanup
- webhook_events terminal-state batched bounded cleanup
- geo_restriction_logs batched bounded cleanup
- scrape_results batched bounded cleanup
- retention single-batch short-transaction deletion skeleton
- RetentionBatchPolicy bounded-deletion parameters
- retention batching/timeout/mutex configuration section
- bootstrap admin key switched to 0600 file delivery (aligned with the Jenkins/GitLab pattern)
Bug Fixes
audit_serviceerror mapping:RepositoryError::DatabaseErrorhas two states (platform rebuildsDbErr::Custom, the lightweight side usesDatabase(String)), preserving theDatabasevariant semantics- i18n locale test environment race condition (
CRAWLRS_LANGprocess-level env serialized via ENV_MUTEX + explicit cleanup) - rustdoc 1.98 strict mode: 88 cases (bare links converted to code font/full paths, lazy continuation indentation, bare generic HTML escaping)
- clippy 1.98: explicit
result_large_errexemption,cmp_ownedeliminated - sticky TOCTOU race-condition test window 1ms→50ms, eliminating intermittent failures under CI load
- Merge remote-tracking branch 'origin/main': unified the dual-remote baseline (after the rebase, same-named fixes landed on gitee first)
- Fixed all red legs in CI/Release — deny/matrix/acceptance/sdk/MSRV
- Fixed all red legs in CI/Release — deny/matrix/acceptance/sdk/MSRV
- 1.98 toolchain fixes: doc links/HTML escaping/.locale test race condition/result_large_err (88 cases)
- RepositoryError::DatabaseError two-state mapping preserves the Database variant semantics
- Added the missing redis 1.7.0 sha1_smol entry to Cargo.lock
- Enqueue outbound crawl links before the adaptive-stop determination, fixing depth>1 unreachability
- Dual-layer SSRF protection for outbound webhooks + closed out strix/OCR audit leftovers
- Removed hardcoded leaked credentials from test_scenarios.py (CWE-798/CWE-540)
- Unified JSON extraction rejection and authentication errors in the ApiResponse wrapper
- scrape route assembly injects an Option-shaped Extension, fixing all POST /v1/scrape 500 errors
- wait_audit_tasks tolerates cancelled tasks, so shutdown waiting no longer panics
- ShutdownCoordinator::wait_for_completion only waits for the trigger signal, fixing the worker's 30s self-termination
- Review-issue fix batch + e2e suite hardening + test reinforcement
- Fixed the garrison_listener flaky test's global-state race condition
- Added skip guards to handler/middleware DB-dependent tests
- Added skip_if_no_test_db guards to DB-dependent tests
- remove needless PoolConfig::default() update
- Closed out verification issues on the latest main — workers geo gating regression + acceptance/webhook lint (full-matrix re-verification passed)
- Completed audit/geo policy parameter call sites + acceptance fmt
- webhook signature unit tests switched to the free function generate_signature (the method was removed with the msg_id fix)
- fix(webhook)+test(acceptance): msg_id consistent with the webhook-id header (signature-verification defect) + webhook/SDK/extract in-depth acceptance scenarios
- Completed retention mock signatures under the test-mocks/unit combination + unix-only test platform gating
- Added a kind=path annotation to the /v1/crawl/{id} path parameter (sdforge infers GET parameters as Query by default)
- Carried the SSRF disable switch through SsrfValidator::validate and the engine layer
- e2e scripts changed from hardcoded API keys to environment-variable injection
- converge reinforcement — API-shape scheduling + configuration lower-bound validation + test assertions
- testcontainers max_connections 2000 eliminates intermittent parallel pool-creation timeouts (addressing the flaky root cause)
- Fixed clippy/fmt gates after the stable toolchain upgrade
- Added the dbnexus dependency to the auth side (the garrison_interface bridge needs DbPool; fixes CI Feature Matrix auth/teams-only compilation)
- Explicit minimal permissions for the check-tech-debt workflow (CodeQL missing-workflow-permissions)
Other Changes
- All in-house dependencies switched to the crates.io registry: confers/inklog/dbnexus/sdforge/oxcache/limiteron → 0.x.0-rc.5, trait-kit → 0.5.0-rc.6, garrison → 0.9.0-rc.2, vecboost → 0.3.0-rc.1; removed all
../baseand../garrisoncross-repository path dependencies and the[patch.crates-io]section, so CI no longer depends on sibling-repository checkouts - Merged the forked histories of the github/main dual remotes (dual-remote documentation alignment, BDD acceptance-line material retained in history)
- Added the 0.2.0-rc.2 section (Version Consistency gate)
- unify codeql-action to 4.38.0 (supersedes #36/#37)
- bump github/codeql-action/init from 4.37.7 to 4.38.0
- Merge remote-tracking branch 'github/main'
- All in-house dependencies switched to the crates.io registry; bumped to 0.2.0-rc.2; incorporated parallel in-flight changes
- Corrected the test-matrix database description from "only supported backend" to three-backend support
- Corrected the architecture documentation's database-backend description from PostgreSQL-only to three-backend support (PostgreSQL/SQLite/MySQL)
- Cleaned up trailing whitespace and end-of-file newlines
- Added a typos allowlist to eliminate false positives
- Integrated the detect-secrets baseline
- Integrated pre-commit gates (formatting/clippy/check/private-key scanning)
- Cleaned up redundant old copyright headers in test modules and completed CODEOWNERS/fixtures attribution
- Synced README/FAQ attribution years and di directory copyright headers
- Completed the copyright line in the LICENSE header (Apache-2.0 dual copyright notice)
- validator converged to the sdforge::validator re-export (incorporating parallel-session changes)
- sea_orm integrated via the dbnexus re-export (incorporating parallel-session changes)
- cargo fmt (formatting fixes required by the agentstem workspace hook)
- Existing maintenance changes from parallel sessions (CI/documentation/dependencies, etc., outside the i18n scope)
- Corrected the description of Bloom gating in the interner comparison table (this crate has no bloom feature)
- Incorporated comment-cleanup changes (before merging back to main)
- include_patterns seed switched to en.wikinews.org + added a tag-stripping extraction case
- Completed the CrawlConfigDto.max_pages test-fixture literal
- Finished migrating engines traits shim references (unit-test imports corrected to canonical paths)
- Removed the backward-compatibility layer, unified on the latest code paths
- include_patterns seed switched to en.wikinews.org + added a tag-stripping extraction case
- E2E/integration tests all switched to real news sites, example.com prohibited
- Recorded verification of leaked-key invalidation and closeout of outstanding items
- Ignored the api_test.sh smoke-test output artifact test_results.txt
- Incorporated in-flight R-key annotation cleanup (19 files, comment-only changes)
- Test-quality hardening report (14 defect fixes + coverage matrix + audit notes)
- Semantic-level API E2E suite and live orchestration (Stage 6 liveapi)
- Fixed LimiteronService signature drift and serialized shared-library queue semantics tests
- README hero v2 redesign (removed redundant spacing/eliminated tagline duplication/fixed an orphan character in the subtitle)
- Documentation system updated
- Fully synced dependencies to the latest and normalized minor-level version notation
- Removed the unused webhook_repo field from CrawlUseCase
- Removed BOOTSTRAP-DIAG debug code
- Removed the outdated tls_fingerprint_check example
- Unified CapturingLogger as a shared test_utils module
- Removed the regex_cache dead-code propagation chain + fixed allow(dead_code)
- align local crate version constraints with rc.1 base crates
- Unified the offline filtering mechanism (sdk dual tagging) + retries guard + CI aligned with env offline mode
- Added a scheduled cleanup workflow to delete leftover dependabot/* branches
- cargo fmt (webhook_service signature block formatting)
- Acceptance coverage updated
- Added the map unauthenticated 401 scenario + linked the benchmark comparison in EN README (converge)
- docs+ci(benchmark/guide/acceptance): linked the comparison report, acceptance usage instructions, CI acceptance job
- Full BDD acceptance-scenario coverage (8 features, 40 scenarios) + made the SSRF switch effective + sdforge path annotation fix
- Ignored the .workbuddy-ai/ workspace directory
- Cucumber scenarios and wiremock support layer for the six major business endpoints
- cargo fmt (map test assertion line b...
Release v0.2.0-rc.1
This release upgrades dependencies github/codeql-action/analyze, github/codeql-action/autobuild, github/codeql-action/init, etc., adds scrape cache TTL decoupled from search and a retention cleanup scan index, fixes the sticky TOCTOU race-condition test TTL 1ms→50ms… and the rate_limit_helper result-large-err allow…, and adds the library crate not tracking Cargo.lock… and the CHANGELOG 0.2.0-rc.1 section (release gate).
Features
web-axumfeature: the auth-off web middleware build surface (platform implicitly enabled auth, so this path was not buildable in the original feature matrix; it can now be built/tested independently)allow_unauthenticated_protected()explicit opt-in (auth-off surface)- scrape cache TTL decoupled from search
- retention cleanup scan index
- Registered RetentionWorker and injected the cleanup data source
- RetentionWorker unified retention-period cleanup
- webhook_events terminal-state event cleanup interface
- geo_restriction_logs retention-period cleanup interface
- scrape_results retention-period cleanup interface
- Added a retention retention-period configuration section
- All in-house dependencies switched to local paths (dbnexus/inklog/sdforge/limiteron/garrison/trait-kit/oxcache/confers)
- add max_results config to cap API limit
- add bootstrap service for initial admin setup
- rewrite SmartSearchEngine as concurrent aggregator with RRF fusion
- add dedup and rrf modules for smart engine refactor
- add Exa/Parallel/Tavily fallback search engines
- remove Google search engine
- add locale/country support to all search engines
- Added a readiness probe and an in-process event bus
- T093 integrated KG structural-hole signals into the frontier scoring pipeline
- T059 MLLM autonomous-navigation crawl example
- T072-T087 frontier algorithm exploration
- T061-T071 Prometheus metric instrumentation + test coverage + benchmark extension
- T045-T060 MLLM autonomous-navigation crawl engine
- Phase 1 TLS fingerprint-spoofing engine — wreq BoringSSL backend/JA4 fingerprint
- Phase 0 security hardening — shutdown timeout fix/SSRF default deny/startup security checks
- Added task-management API examples (task query, batch cancel, synchronous wait)
- Completed rate-limiting examples (basic rate limiting, circuit breaker, team concurrency control)
- Completed proxy examples (HTTP proxy, proxy authentication, proxy rotation)
- Completed extract examples (multi-rule extraction, output formats, XPath extraction)
- Completed cache examples (cache configuration, oxcache integration, TTL management)
- Completed auth examples (API Key, Bearer Token, scope validation, team isolation)
- Completed advanced examples (async batch, streaming, error handling, metrics export)
- add extraction_prompt and extraction_schema support
- only_main_content body extraction integration (T074/R-content-001)
- T028 retry classification + T067 adaptive stop condition
- standardwebhooks signatures + hickory-resolver DNS
- Implemented a pgmq-style message queue (repository pattern)
- integrate locale-aware messages in handlers
- localize error messages and response builder
- add Accept-Language middleware and route registration
- Added the Mozilla Fluent i18n system
- Configuration/i18n/infrastructure improvements
- Stage 4 WaitFor + Hedge integration (T069-T070)
- T068 TabPool reuse layer (DashMap + CAS LIFO stack)
- T066 should_crawl delegates to FilterChain + extract_and_queue_links uses Frontier
- T065 Frontier priority queue + domain round-robin
- T064 URL scorer trait + CompositeScorer + keyword/path-depth scorer
- T063 URL filter trait + FilterChain + three concrete filters
- Stage 3 closeout T057-T062 advanced cache modes + MRT waterfall timeouts
- Stage 3 P0 proxy rotation pool and ProxyProvider abstraction (T054-T056)
- Stage 3 P0 body extraction and URL layered dedup (T044-T053)
- Stage 2 scheduling infrastructure and identity rotation (T020-T042)
- Stage 1 P0 core protections and scheduling (T008-T019)
- Stage 5-7 + T034 full verification + pre-commit review fixes
- T030 — migration 005 marks the old api_keys/scopes tables as deprecated
- Stage 6 — API Key issuance + old-table deprecation + re-signing script (T027-T029)
- Stage 5 — CrawlrsAuditListener bridges garrison events to the crawlrs audit service (T024-T026)
- Stage 4 — removed hand-written authentication dead code + security/performance fixes (T018-T023)
- Stage 3 — middleware bridging and garrison RBAC rewrite (T012-T017)
- Stage 2 — DI initialization and BootstrapError::Auth (T010-T011)
- Stage 1 — garrison infrastructure setup (Config/DAO/Interface)
- Stage 0 — prerequisite gate verification + garrison dependency introduced (T001-T003)
- Stage 4+5 — webhook gating + feature matrix/CI/docs acceptance (T023-T035)
- Stage 3 — rate-limit feature gating and NoopRateLimitingService (T017-T022)
- Stage 1+2 — auth default-identity and teams single-tenant degradation (T006-T016)
- Stage 0 — feature scaffolding and default-identity constants (T001-T005)
Bug Fixes
exe_pathcross-platform naming (incorrect_exe_pathreference on the Linux side)- Missing test construction for three new
WorkerManagerDepsfields (added minimal test doubles) - clippy
-D warnings:result-large-err / struct update no-op / unused variable - rustdoc link debt (crate-level lint declaration)
- cargo-deny: lru upgraded to 0.18.2; paste transitive-dependency exemption recorded (RUSTSEC-2024-0436/2026-0258)
- sticky TOCTOU race-condition test TTL 1ms→50ms (the in-lock reselect window was too narrow, causing threads to reselect different URLs after expiry)
- Added the missing rate_limit_helper result-large-err allow
- Gate closeout — three clippy -D warnings items (result-large-err/struct update noop/exe_path unused), deny exemption for paste, lru upgrade to 0.18.2
- wreq declared as a 0.16 placeholder (upstream 5.x all yanked/6.0-rc broken, resolve gate); adaptation recorded
- cargo-deny with explicit --features full (isolating the wreq 5.x upstream all-yanked incident) + rustfmt normalization
- Gate fixes — lib cross-platform exe_path naming, rustdoc link-debt lint declaration, WorkerManagerDeps three-field test doubles and cleanup_expired completion
- Added --allow-dirty to cargo publish (so Cargo.lock resolution updates do not block the release)
- workspace.package version synced to 0.2.0-rc.1
- fix(auth-off)+test(purify): protected routes default to 401 + anonymous restricted-identity opt-in (T037/T038), Noop made explicit (T039), sdk integration-layer duplicate cases purified (T041), web-axum feature unlocks the auth-off build surface
- Local path dependencies unified to relative paths (cross-platform fix)
- dbnexus-macros local path (eliminating the proc-macro-error2 warning pulled in by registry 0.4.4)
- Eliminated unused_mut/unread-assignment warnings (final_content lazily initialized as immutable)
- Route registration gated by the http feature + chromiumoxide 0.9.1 API adaptation + ssrf migration test fixes
- resolve 30 bugs from OCR deep scan report
- align CacheWriter::set signature with oxcache 0.4 trait
- resolve compiler warnings in webhook and wreq engine
- fix compilation errors from module renames and API changes
- correct trigger function name in message queue schema
- use concrete repo impls for v2 Extension type matching
- replace SSRF host rewrite with resolve override
- inject FlareSolverr timeout from config, unify fallback URL
- remove weak password defaults from docker and test configs
- Fixed the crawl example's CrawlConfigDto missing extraction_prompt/extraction_schema fields
- allow dead_code lint for sdforge forge macro
- Fixed the proxy_pool sticky TOCTOU race condition + worker timing flaky test
- Fixed pre-existing test compilation errors (import paths, trait signatures, type mismatches)
- Removed the invalid exclude field + cleaned up configuration
- handlers/middleware fixes and optimizations
- Fixed tests/main.rs test-target compilation errors and garrison singleton contention
- Fixed HIGH/MEDIUM issues found in the subagent review
- Merge dev/v0.2.0: three-dimensional review HIGH/MEDIUM fixes + documentation sync
- Three-dimensional review HIGH/MEDIUM fixes + documentation sync
- T067 registered the adaptive module + hedge reset warning fix
- T001+T002 security configuration hardening
- Fixed the SSRF IPv4-compatible IPv6 bypass vulnerability and strengthened test coverage
- Explicitly specified the tool parameter for cargo-llvm-cov
- Added a User-Agent header to crates.io API requests to avoid 403
Other Changes
- auth-off security semantics tightened: protected routes deny by default (401), removed the
DEFAULT_IDENTITY_TOKEN_HASHplaceholder hash andfull_access()identity injection; after opt-in, an anonymous restricted identity is injected (denied()scope, no token_hash) NoopRateLimitingService/NoopWebhookServiceassembly logs raised to one-time error level (making the security-semantics degradation explicit)- sdk route tests converged to the unit layer (removed semantically duplicate cases between tests/sdk_api_test.rs and src/presentation/sdk/tests.rs)
wreqdependency declaration changed to a0.16placeholder (upstream incident: 5.x all yanked, 6.0.0-rc broken builds; enablingengine-tls-fingerprintrequires adaptation to the 0.16 API)- protected routes default to 401 under auth-off (embedders that relied on all-open unauthenticated access must opt in explicitly)
- Library crate does not track Cargo.lock (CI publish resolution refresh makes it dirty)
- CHANGELOG 0.2.0-rc.1 section (release gate)
- Reverted --allow-dirty (no bypassing the dirty gate)
- Cross-repository dependencies made version-only + local [patch.crates-io] redirection (CI release mode)
- bumped crawlrs to an rc version (crates.io +0.1 minor prerelease)
- auth-off default 401 + opt-in new semantics
- Eliminated clippy warnings (field_reassign_with_default/redundant_pattern_matching/unused import, etc.)
- DbPool test const...
Release v0.1.0
This is the repository's first release; there is no earlier release to compare against. The full commit history is at: https://github.com/Kirky-X/crawlrs/commits/v0.1.0.
Features
- Web platform and API: added handlers and improved middleware in the presentation layer (c682883), unified the API path prefix (aedb264), generated OpenAPI documentation with utoipa (a84ebf6), and unified the API response format and added a timestamp field (330a837)
- Multi-engine crawling: reqwest engine (7f41463), conditional compilation of the Playwright engine (6d0b7db), FlareSolverr engine supporting smart routing to bypass CAPTCHAs (8699679); EngineClient unified public API (d7efb71) and ScrapeOptions advanced capabilities (14fffda); HTTP proxy support for all engines (53633b8); automatic browser download (1de2c48)
- Smart search: unified search module with security hardening (42d5b05), SmartSearchEngine router and factory (9f7bca4), rate limiting/timeout control and test-data loading (531596d)
- Tasks and queues: task state machine and dedup optimization (7ee4008), QueueClient unified queue interface (31f2f8f), PostgresTaskQueue (a1cf7a1), automatic task expiry and access control (1fe2446), improved retry strategy (8d38ec1)
- Access control and teams: API key scope management, feature flags, audit logs (6f17710), persistent team-level domain blacklist (720b413), geographic access restrictions wired to the database repository (deea68d), geographic-restriction data table (2d4443c), improved IP access control and multicast IP blocking (e21335a)
- LLM extraction: multi-provider support and extraction enhancements for the LLM service (f86f543), prompts.toml prompt configuration (3d88e36), local Ollama model configuration (c276eda)
- Storage and cache: storage abstraction layer and multiple storage implementations (fee3d75), S3 object-storage integration (7c46dfb), SQLite support and a performance benchmark suite (29a7d04)
- Underlying library unification: confers fully took over configuration (287b47b), dbnexus fully refactored database operations (7149827), limiteron refactored rate limiting/concurrency control/quota management (821ba45), inklog fully replaced tracing (19996de), sdforge interface wrapping (999c8e4), CacheService trait and OxcacheService (af628bb)
- Other core capabilities: text-encoding processing module (24c0d08), smart port-sniffing module with default port 8899 (d09e3a7), degradation strategies and alert rules (b29fc62), completed WebhookManagement/TeamManagement/ScrapeService domain services (361bf80), webhook management and worker (6b5ad3c), environment-variable validation and infrastructure modules (988fd79), completed database/config/sdk examples (2ed1998)
- Engineering infrastructure: criterion benchmarks (a5d1900), Docker test environment (004d55b), feature test matrix (9bea9b9), Python API test framework (a667f6e), complete CI/CD harness configuration (47b5a94)
- Fixed 3 strix vulnerabilities + configured the complete CI/CD harness
- Completed database/config/sdk examples (T024-T026)
- T049 follow-up — removed the testcontainers redis dependency
- T049+T047c+T044 — completely removed the Redis runtime dependency
- T047+T048 — migrated RobotsChecker to CacheService and ScrapeWorker to TeamSemaphore+DashMap
- Defined the CacheService trait and implemented OxcacheService (T045+T046)
- Updated bootstrap module initialization to trait-kit (T040)
- AppState built via AsyncKit, removed state_manager (T039)
- Implemented trait-kit module definitions and AsyncAutoBuilder (T037+T038)
- T020-T022 completed domain services — ScrapeService + WebhookManagement + TeamManagement
- T016-T019 queue industrialization — MessageQueue trait + InMemoryQueue + ApalisQueue + adapters
- Phase 7 environment-configuration improvements — CI/Docker/env (T022-T024)
- Phase 6 feature-gating improvements — gated optional dependencies + cfg guards (T020-T021)
- Phase 3 sdforge interface wrapping (T014-T016)
- inklog fully replaces tracing (Phase 2: T009-T013)
- Migrated local path dependencies to the latest crates.io versions
- Added the as_inner method and a database session wrapper
- update create_pool and add Default impl for DatabasePool
- add payload limit extraction for tasks
- add crawl use case tests
- merge test enhancements from worktree
- Added rate-limiting service and task state-machine test files
- add task queue tests
- Optimized the code structure
- Continued handling user requests
- Refactored rate limiting, concurrency control, and quota management using the limiteron library
- Fully refactored database operations to use the dbnexus library
- Added inline default-value support
- Used the confers library to fully take over configuration management
- complete project refactoring with confers config migration
- Added a timestamp field to the unified API response format
- Added automatic browser download
- Optimized dependency imports and reduced binary size
- Added conditional-compilation guards for the Playwright engine
- Merge branch 'feature/modernization-2026-01'
- Phase 3 DTO, use-case modules, and domain-event mechanism
- Phase 1-2 infrastructure and API standardization
- add utoipa for OpenAPI documentation
- Added new infrastructure components
- Enhanced the LLM extraction example
- add configuration service and documentation
- improve configuration structure and validation
- Added authentication rate limiting and DTO validation
- Added a pre-commit check script
- Split error types into a standalone module to improve code organization
- Unified constant management and added 15+ key constants
- update all Rust dependencies to latest versions
- add environment variable validation and infrastructure modules
- migrate CreateScrapeUseCase from EngineRouter to EngineClient API
- Configuration externalization and error-handling improvements
- add SafeUrl type and validated_url field to DTOs
- add code review report and secret handling utilities
- add dependency injection module (shaku)
- update engine router demo with enhanced test scenarios
- enhance LLM service with multi-provider support and improved extraction
- enhance LLM extraction test with markdown and JSON formats
- add prompts.toml and update LLM settings
- Moved engine configuration from hardcoded values into configuration-file management
- Added FlareSolverr engine support implementing smart routing to bypass CAPTCHAs
- Fixed the Sogou search parser's URL-parsing issue
- Add .sisyphus directory to .gitignore
- Add AGENTS.md files to .gitignore
- Add .claude and .shared to .gitignore
- Added project documentation and a logo
- add new handlers and improve middleware
- add HTTP proxy support for all scraping engines
- Completed a comprehensive production-environment test plan
- Added test-runner scripts and documentation
- Added the feature test-matrix configuration
- Added a complete API test suite
- Added a Python API test framework
- Added a complete Docker test-environment configuration
- Added a Docker test environment and headless Chrome support
- optimize critical path performance
- add API key scope management, feature flags, and audit logging
- Extend ScrapeOptions with advanced capabilities
- Add engine critical issues fix proposal
- Implement EngineClient as unified public API
- refactor config module into sub-modules
- add unified QueueClient with proper validation
- add unified search module with safety hardening
- Add project documentation and test utilities
- Add test data support to Sogou search engine and enable search engine tests
- Enhanced HTTP handler functionality and error handling
- Improved application startup and worker management
- Enhanced core functionality of task and crawl services
- Enhanced storage service functionality and error handling
- Improved the robustness and configurability of the rate-limiting service
- Added the crawlrs API and worker services to docker-compose.test.yml
- Added benchmarks for core system functionality
- add Google search integrationSolverr support
- add browser automation tests with remote Chrome support
- add max_attempts parameter to control port scan range
- add Google search engine with FlareSolverr to test suite
- add rate limiting, timeout control, and test data loading to SmartSearchEngine
- enhance Google and Sogou engines with logging improvements
- improve logging format and assertions
- enhance with rate limiting and anti-crawl measures
- add engine registration and listing methods
- add SearchResult Default implementation and update plan
- add smart search engine router and factory
- include Bing, Baidu, Sogou in test search aggregator
- Completed implementation of degradation strategies and alert rules
- Implemented automatic task expiry and access control
- implement persistent team-level domain blacklist
- integrate database-based geo restriction repository
- implement S3 object storage integration
- update task entity and repository implementation
- enhance crawl and scrape core logic
- update webhook management and worker
- update search engines and router
- implement team management and geographic restrictions
- add geographic restrictions table
- Integrated leading-newline handling into the web content handler
- Added leading-newline removal for strings
- enable only Sogou search engine for "星光盛典" search
- Configured the Google search engine and disabled the others
- Added a text-encoding processing module supporting Unicode detection/conversion, encoding-format detection, encoding conversion, error handling, and performance optimization
- Update task repository implementation and add expires_at migration
- Complete boundary scenario validation (UAT-007 to UAT-012)
- Added a storage abstraction layer and multiple storage implementations
- Added SQLite support and a performance benchmark sui...