Skip to content

Releases

Release v0.2.0

Choose a tag to compare

@github-actions github-actions released this 15 Nov 04:37

First stable release on crates.io. All in-house dependencies unified to stable versions (minor precision), alchemize switched to the crates.io registry, no path/git dependencies remain in this repository, and the cargo publish pipeline is unblocked. This section incorporates the former [Unreleased] section.

Features

  • i18n system rewrite: the messaging layer switched to the sdforge::i18n host registry, locales/*.ftl files are embedded at compile time (src/i18n/catalog.rs, include_str! + line parsing at startup via register_translation), and deployment artifacts no longer ship an on-disk locales/ directory; the key guard was upgraded to bidirectional consistency between the embedded manifest and the on-disk directory (tests/unit/i18n); t_with_args/tr_log_args parameters simplified to &[(&str, String)] ({ $name } replaced by name)
  • crates.io release pipeline: with all dependencies on the registry, cargo publish is unblocked; release.yml migrated to the final family state (OIDC trusted publishing preferred + CARGO_REGISTRY_TOKEN fallback, idempotent creation via a gh release script step)
  • Trial integration of the alchemize html engine (path dependency, converter=alchemize)

Bug Fixes

  • rag-local dead on arrival at runtime: the vecboost dependency with default-features=false stripped out the onnx main engine, while the provider was pinned to EngineType::Candle — the vecboost factory fails fast on an engine that was not compiled, so rag.enabled=true + embedding_provider="local" failed at startup. Fix: set features=["onnx"] explicitly on the dependency and pin the provider to EngineType::Onnx
  • Two needless_bool cases under nightly clippy: the sitemap seed guard now returns the feed result directly, and the cache_mode POST check uses a negated match expression
  • SDK integration test E0061: I18nBundle::load in sdk_api_test changed to a two-argument call (a stale test missed during the switch to sdforge host registration)
  • acceptance teardown single_match: logger shutdown in tests/acceptance/teardown.rs changed to if let (the release workflow's clippy --tests would flag it)
  • BDD infrastructure: closed out two accompanying defects — swallowed warn logs and a Ryuk container leak
  • Closed out two release-gate test defects — acceptance teardown clippy single_match and the Python create-scrape assertion of 201
  • sdk_api_test's I18nBundle::load changed to a two-argument call — a stale test missed during the switch to sdforge host registration (root cause of CI integration leg E0061)
  • fmt gate changed to target two packages with -p (--all affected inactive path dependencies)
  • coverage job narrowed to --lib (90.12%; a cold build of all targets timed out)
  • Coverage timeout increased 45→75 minutes (measured 44m25s for a cold compile after dependency changes; cancelled jobs do not write back the cache)
  • deny switched to run directly on the runner + CodeQL added sibling checkout (the Docker action cannot see symlinks)
  • Added vecboost to the sibling checkout chain (a path dependency of alchemize)
  • All cargo jobs check out the alchemize sibling (needed during path dependency resolution)
  • Integration Tests timeout increased 25→40 minutes (measured 25m15s, a borderline timeout that was cancelled)
  • Coverage job timeout increased 30→45 minutes (measured 30m16s, a borderline timeout that was cancelled)
  • Closed out two accompanying infrastructure defects — swallowed warn logs and a Ryuk container leak
  • Adjudication reason enum aligned with API literal values (including spaces)
  • CodeQL false-positive adjudication switched to PATCH (the current REST endpoint method)

Other Changes

  • Markdown converter default switched to alchemize: the default of [markdown].converter changed from moli to alchemize (in-house html engine: htmd base chain + in-house deep chain, with built-in empty-output safeguards, honest degradation, and math $TeX$ fidelity routing); the content feature absorbs alchemize, and htmd is retained as a pass-through diagnostic arm. Equivalence measured: 13 cases (5 real pages + 8 synthetic), 10/13 byte-identical (only trailing newlines differ); math pages improved from leaking MathML to $TeX$; 2 navigation-element heuristic differences (no content loss); throughput on wiki 1.06MB×20: moli 105.6 vs alchemize 96.8 MB/s (-8%); assembled-chain output byte-for-byte identical to alchemize native on 5 real pages
  • MSRV 1.97 → 1.97.1: aligned with alchemize's rust-version; CI toolchain matrix and README/FAQ/CONTRIBUTING text synchronized
  • Coverage job narrowed to --lib: after alchemize entered the tree, cold builds of all targets repeatedly exceeded 75m (cancelled jobs do not write back the cache); lib-only measured 90.12% (integration tests added <0.01pp), with the 80% gate unchanged
  • CI gate criteria: fmt changed to target two packages with -p crawlrs -p crawlrs-examples; coverage/integration/BDD job timeouts raised per measurements (30→45→75 / 25→40 / 30 minutes)
  • All in-house dependencies unified to stable versions (minor precision): confers 0.6, dbnexus 0.6.1, garrison 0.9, inklog 0.3, limiteron 0.3, nebulaid 0.3, oxcache 0.5, sdforge 0.5, trait-kit 0.5, vecboost 0.3, fluxweir 0.1 (resolved 0.1.2), alchemize 0.1; rc pins on direct dependencies cleared to zero
  • alchemize switched to the crates.io registry: removed path = "../alchemize", so cargo publish/cargo package are no longer rejected due to a path dependency; the CI .github/actions/alchemize-sibling checkout chain was removed (a clean checkout resolves directly)
  • fluxweir returns to the registry: from 0.1.1 it provides with_pool/build_app_with_pool/TrustedIdentity, so the rev-pinned git dependency was removed and deny's allow-git cleared
  • moli-html2md dependency and conversion chain retired: deleted moli_adapter.rs / moli_converter.rs / converter_chain.rs (the moli-primary + htmd-fallback chain), removed dep:moli-html2md and dep:ego-tree from the content feature; the crawlrs_md_converter_fallback_total metric was removed along with the chain (alchemize's built-in fallback is no longer counted separately)
  • fluent-bundle dependency retired: zero references after the messaging layer unified on sdforge::i18n host registration; the I18nSettings.locales_dir field was removed as well (the CRAWLRS__I18N__LOCALES_DIR environment variable is no longer read)
  • Migration note: CRAWLRS__MARKDOWN__CONVERTER=moli (the old default) is rejected during configuration validation (startup fails); it must be changed to alchemize (the default) or htmd
  • security workflow permissions: added checks: write — rustsec/audit-check needs it to write back the check-run (R-17: Resource not accessible by integration)
  • CodeQL alert adjudication workflow: made false-positive adjudication idempotent (skips already-adjudicated alerts), aligned the adjudication reason enum with API literal values, and switched to PATCH (the current REST endpoint method)
  • Integration Tests timeout 40→70 minutes — a ~45m cold compile after a dependency lockfile change caused job 7e64a2b to be cancelled
  • RUSTSEC-2025-0141 bincode unmaintained: same-source exemption in two files
  • 0.2.0 stable release wording — new CHANGELOG section, README/architecture/API/guide versions and dependency baseline aligned
  • In-house dependencies unified to stable versions + version bumped to 0.2.0 — minor precision, alchemize switched to the crates.io registry
  • release.yml migrated to the final family state — OIDC trusted publishing + token fallback, gh release script step; removed the alchemize sibling checkout
  • Cleared two nightly clippy needless_bool cases — the sitemap seed guard returns the feed result directly, and the cache_mode POST check uses a negated match expression
  • Added checks: write — rustsec/audit-check needs this permission to write back the check-run (R-17: Resource not accessible by integration)
  • Unified interfaces across the crate family — trait-kit rc.14 (PluralCategory re-export) + sdforge rc.11 + confers rc.13 (as involved in this repository), with no breaking changes to the public API
  • Removed the dead fluent-bundle dependency (the messaging layer already unified on sdforge::i18n host registration, zero references)
  • Rewrote the in-house Fluent i18n system + pinned the vecboost onnx engine + batch of engine/worker fixes
  • pre-commit cargo-fmt hook synced to the -p criteria, closed out remaining documentation items
  • Removed the vecboost sibling checkout (audio cannot be enabled; verified that resolution succeeds without it)
  • bump actions/upload-artifact from 4.6.2 to 7.0.2
  • bump actions/download-artifact from 5.0.0 to 8.0.2
  • bump zizmorcore/zizmor-action from 0.7.1 to 0.8.1
  • Test statistics re-measured after the moli chain removal (6,548 / 229,992 lines)
  • Removed moli-html2md; the alchemize html engine becomes the default converter
  • Synced with upstream feature shape (one feature per format; html includes the htmd base chain + deep chain)
  • fluxweir switched to crates.io 0.1.1 (deny allow-git removed that source as well)
  • Made the false-positive adjudication workflow idempotent (skips already-adjudicated alerts) and removed temporary diagnostics
  • Added 404 diagnostic output to the CodeQL false-positive adjudication workflow
  • Added a CodeQL false-positive adjudication workflow (identifier-logging FPs, with security-events permission delegated)

Related PRs / Issues

  • #45 — bump actions/download-artifact from 5.0.0 to 8.0.2
  • #46 — bump zizmorcore/zizmor-action from 0.7.1 to 0.8.1
  • #47 — bump actions/upload-artifact from 4.6.2 to 7.0.2

Release v0.2.0-rc.4

Release v0.2.0-rc.4 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 10 Nov 16:01

This release fixes the depth counter clamped at a lower bound of 0 and intermittent BDD connection closed issues; acceptance is pinned to offline mode in release tests for consistency with CI.

Bug Fixes

  • CI fixes: six no-default warnings across the five Feature Matrix combinations (RetryReason cfg gating, unused parameters in race/sequential, and three url_redaction cases attributed to features via allow); the engine-moli memory-gating test's cross-submodule access changed to state_for_test() (also fixing the _available_mib warning); the BDD job added the PostgreSQL service and migrations steps (previously TEST_DATABASE_URL pointed to a nonexistent service, so bootstrap failed wholesale with "Connection closed")
  • Queue metric fix: PostgresTaskQueue depth counter clamped at a lower bound of 0 — crawl subtasks are enqueued via a direct repository.create write while workers decrement via queue.dequeue, so negative drift caused u64 wraparound: in debug builds a subsequent enqueue overflowed and panicked (the root cause of intermittent BDD "connection closed before message completed"), while release builds silently emitted garbage crawlrs_queue_depth; includes a regression unit test
  • Security channel: added same-source exemptions in .cargo/audit.toml and deny.toml (including RUSTSEC-2026-0459, where sqlx 0.9's <0.38 constraint blocks the fixed version 0.39); adjudicated all 18 open CodeQL alerts down to zero (9 identifier/masked-log FPs, 2 high-entropy key-digest FPs, 2 per-request TLS option won't-fix, 5 test-fixture used-in-tests)
  • Comments and documentation: cleaned up task-number/review-stamp comments (in CI and worker/engine/handler comments); fixed 3 doctest errors (record_block()/intercepted_count()/HtmdMarkdownService paths); aligned API_REFERENCE/FAQ/USER_GUIDE versions and health examples; re-measured README statistics (230,418 lines)
  • Examples: 18 examples previously always failed because EngineClient::new() registered zero engines; added crawlrs_examples::default_engine_client() (ReqwestEngine) and wired it into 17 examples (mllm_scrape requires full LLM DI, so it deliberately keeps an explicit zero-engine setup to avoid ordinary HTTP success masquerading as MLLM success); basic_scrape measured HTTP 200 on example.com; all examples compile with -D warnings
  • Dependencies: html-escape 0.3.1, toml 1.2.1, uuid 1.29.2 (cargo-outdated shows only testcontainers-chain constrained items, which remain pinned)
  • Verification: lib 6379/0, integration 170/170, main 790 passed/16 ignored, sdk 8/8, doctest 10/10, line coverage 90.13% (pipeline Step 9 target ≥90 met; CI threshold remains 80%), E2E matrix/static/unit/integration/bench/liveapi six stages PASS (report aggregation SKIPPED by design), BDD acceptance run 5×59/59 consecutively, clippy(default/full/agent-lib)+doc+deny+audit all green
  • Clamped the depth counter at a lower bound of 0, fixing the root cause of intermittent BDD connection closed
  • Pinned acceptance to offline mode in release tests, consistent with CI
  • When git dependencies are present, GitHub Release uses the tag source archive and skips crates verification
  • git dependency detection changed to word-boundary matching with an inline-table form
  • When git dependencies block crates.io, skip explicitly and allow GitHub Release

Other Changes

  • Release 0.2.0-rc.4

Release v0.2.0-rc.3

Release v0.2.0-rc.3 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 10 Nov 12:44

This release adds a Markdown converter chain — moli-html2md primary + htmd fallback — and a WaitFor response-level wait signal (absorbing moli --wait-response…); it fixes and closes out three-dimensional review MEDIUM/LOW items and blacklist/terminal-state invariants, closes out terminal-state reconciliation, the blacklist execution chain, and SDK crawl creation, and adds the E2E matrix, keeps temp live-test artifact directories out of version control, and moves FEATURE_MATRIX out of the repository.

Features

  • Markdown converter chain — moli-html2md primary + htmd fallback
  • WaitFor response-level wait signal (absorbing moli --wait-response-* semantics)
  • MoliEngine — a JS rendering engine that controls an external moli serve via CDP
  • Memory-watermark scheduler and js_degraded degradation flag (engine-moli feature)
  • fluxweir gateway integration — unified exit for LLM/embedding/reranking with single-pool sharing
  • Completed key registration + closed out detection/fallback semantics
  • Made repository error classification explicit (rate limiting / optimistic lock conflicts)
  • Performance baseline regression gate and status alignment — bench.yml/bench_gate.py, PERFORMANCE/CHANGELOG/README updates (T013-T017)
  • Redis shared cache — [cache] backend configuration, fail-closed validation, and memory/redis dual-backend switching (T010-T012)
  • GET /v1/tasks/{id}/ws real-time task status push (T006-T009)
  • Supply migration directories at runtime by dialect and wire up auto_migrate (T004/T005)
  • Three-dialect schema supply — separate migrations/{postgres,sqlite,mysql} directories + consistency tests
  • merge: feat/searxng-meta-search — SearXNG meta-search integration
  • SearXNG meta-search integration — public instance lifecycle and failover
  • Hand-written provider stack retired — LLMServiceTrait established as the injection boundary
  • Embedding unification baseline aligned with the rc.9 line + embed-probe gate locked in
  • Ledger fully cleared — single error-response shape / full fingerprint coverage / result invariants / backend verification
  • Playwright stealth fingerprint consistency upgrade — Sec-CH-UA / dual word lists / New Headless
  • IdModule algorithm validation and auth-off operational logs now emitted via tr_log
  • ID module configuration validation and auth-off degradation logs wired into Fluent
  • Engine health monitoring absorbs garrison HealthCheck/Registry (auth-gated)
  • status_code 17-arm mapping moved down into the sdforge unified table
  • 42 user-visible strings/logs wired into Fluent (en default + zh key alignment)
  • Vision calls support custom OpenAI-compatible endpoints + live-test target corrected (F12)
  • AIMD concurrency target team gauge and increment/decrement debug logs
  • Character-level provenance alignment for extraction results + MLLM hybrid AX-tree snapshots
  • Extraction degradation-ladder quality gate + LLM token-window chunked parallel merge
  • Playwright XHR/Fetch background response capture
  • sitemap streaming as a first-class request source + soft-404 fingerprint + seen prewarm delta
  • Failure-scene snapshots (error signature → HTML + screenshot) + AIMD memory-pressure linkage
  • Session identity triplet binding and retirement of error_score scoring
  • AutoThrottle delay convergence + 429 exponential backoff + Crawl-delay dual clocks + stall detection
  • exa/tavily conditional registration + baidu unwrapping + incremental-crawl conditional requests
  • stealth injection adds webgl_overrider (WebGL vendor/renderer fingerprint override + canvas read noise)
  • Browser-pool per-task BrowserContext isolation (context_isolation enabled by default, with an escape hatch on the TabPool reuse path)
  • per-host discipline component (polite rate limiting + 429 cooldown + bare 403 streak → TLS upgrade) + 429 Retry-After parsing wired up
  • robots.txt fetch failure fail-closed fallback (short-TTL failure-flag cache) + respect_robots request-level override semantics locked in

Bug Fixes

  • Closed out three-dimensional review MEDIUM/LOW items and blacklist/terminal-state invariants
  • Closed out terminal-state reconciliation, the blacklist execution chain, and SDK crawl creation
  • Fixed crawl terminal-state counting/matrix construction and opened up all E2E stages
  • MEDIUM/LOW closeout and MySQL migration CI execution verification
  • Added MySQL 010 legacy url_hash upgrade and cleared the test baseline to zero
  • Fixed three-party review HIGH/MEDIUM issues and closed out remaining problems
  • Completed review MEDIUM/LOW remediation (security/IP/migration/performance)
  • Fixed 12 E2E defects and completed three-party review remediation
  • fix(security)+refactor(core): closed out spec convergence-batch-2
  • coalesce keys include the full request identity and tenant, closing content cross-talk
  • Cleared 30 outstanding OCR review items (concurrency/credentials/CI guards)
  • bytes version notation aligned to the minor level (1 → 1.12)
  • Fixed three false-green gates and closed out UA architecture and ws quota defects
  • Assertions keyed by compilation target and locale, with hardcoding removed
  • Promotion probe concurrency cap + map endpoint status-code criteria aligned
  • Added an in-flight gate to number-pool replenishment, eliminating concurrent duplicate replenishment
  • fetch_update changed to try_update (deprecated in rustc 1.99)
  • Restored the mtime redaction fix accidentally reverted by 60743dd
  • Fixed zizmor's nonexistent v0.12.0 and shallow-clone gitleaks no-op, and removed the always-red cargo vet
  • Wired connect_timeout throughout the production client to prevent connections from exhausting the total timeout
  • js-upgrade reroute dead-end degradation — return the raw response when no JS-capable engine is available
  • Ledger cleared to zero — unified 429 wrapping / error propagation / result invariants / credential hygiene
  • Outstanding items converged — WKWebView family data truth / locale / arch + test gates completed
  • id algorithm parsing assertions aligned with NebulaId rc.3 alias reversal
  • Documentation audit defects #1-#4 — migrate subcommand, cancel route, search config consumption, scrape option exposure
  • Restored auth-off default-deny semantics and the web-axum equivalent build surface (documentation audit defect #6)
  • Chunking/provenance wired into the live extraction path + direct-call max_tokens cap and configurable timeout
  • Added the content_extractor field to WorkerManagerDeps test construction
  • Fixed stage-failure PIPESTATUS determination + search fault tolerance allows explicit validation errors through
  • Fixed construction adaptations, outdated assertions, and clippy violations exposed by live gates
  • Unified garrison global-singleton cross-test mutex + RBAC missing-table loudly skip
  • content_extractor quality-gate Facade injected into the worker via DI
  • coalesce waiters copy the leader's result before completing
  • Strengthened Baidu verification-page title detection + search unit tests assemble real engines
  • 429 determination short-circuits before data-JSON
  • Independent profile directory per chromium instance + fallback reclamation on pool destruction
  • Request-level cookie jar preserves Set-Cookie across the redirect chain
  • Direct HTTP replaces EngineClient, so local LLM endpoints are no longer blocked by crawl SSRF
  • Wired up the CRAWLRS_DISABLE_SSRF_PROTECTION bypass
  • Removed the dead jsonwebtoken dependency — zero references across src
  • single-flight waiters in the same batch share the leader's result, eliminating serial amplification of failures
  • HostDiscipline lazy sweeping + toDataURL does not materialize the context
  • gate changed to a per-host reservation schedule, with ≥ min_interval spacing between concurrent requests to the same host within a batch
  • Rolled back tab-return browser-level cookie clearing + added host anti-bot memory regression tests
  • URL normalization strips known tracking parameters + robots matching carries the query
  • Absolute-ize relative links in body HTML + escape line starts in the PlainText degradation branch
  • host anti-bot memory + DomStable degradation + DFS depth key + race escalation reassignment + _abck recovery
  • r2 aggregation chain hardened — per-engine timeout / locale propagation / cross-engine dedup / outbound rate limiting / challenge-determination fixes
  • bootstrap injects a wreq response-body limit + docs synced with audit fixes + typos allowlist for _abck
  • only_main_content Markdown preserves body structure (links/headings)
  • SmartSearchEngine wired to the production API + RRF dimensional normalization + Bing challenge-page detection and link unwrapping
  • Bloom false positives causing permanent missed crawls + startup prewarming + automatic capacity scaling + DFS semantics corrections
  • stealth CDP injection + true networkidle + wait_for wiring + pool proxy validation + tab cookie clearing
  • race mode wired into anti-bot detection + Akamai detection flag corrections
  • All-scheme proxy interception (Proxy::all) + wreq TLS impersonation landed + UA pool Chromium filtering

Other Changes

  • Release 0.2.0-rc.3
  • E2E matrix and temp live-test artifact directories kept out of version control
  • FEATURE_MATRIX moved out of the repository
  • Disabled the memory-watermark scheduler to eliminate machine-dependent intermittent stalls
  • Feature combination matrix and contract errata (FEATURE_MATRIX)
  • Exhaustive scenarios implemented — cancellation/boundary/validation matrix/batch cancellation and webhook cross-talk fixes
  • Sixth wave of the complexity long tail (clear-to-zero round)
  • Wave five closeout — extraction scheduling unification and billing settlement strategy
  • refactor(core)+fix(security): pending items cleared to zero and complexity long-tail batch T004/T005
  • Wave one convergence — refactoring of 43 high-complexity functions + three-dimensional review fixes
  • Cleared 9 outstanding three-dimensional review items (security/performance/architecture)
  • Behavior-preserving refactor of high-complexity paths and split route_se...
Read more

Release v0.2.0-rc.2

Release v0.2.0-rc.2 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 21 Sep 17:11

This release upgrades the github/codeql-action/init dependency, adds unify-rust-i18n crawlrs residual Chinese cleared to zero (T023) and unify-rust-i18n [T012] crawlrs i18n remediation, fixes the sticky TOCTOU race-condition test window 1ms→50ms… and Merge remote-tracking branch 'origin/mai…, and adds the 0.2.0-rc.2 section (Version Consistency gate) and unify codeql-action to 4.38.0….

Features

  • unify-rust-i18n crawlrs residual Chinese cleared to zero (T023)
  • unify-rust-i18n [T012] crawlrs i18n remediation
  • RAG retrieval-augmentation/rerank/embedding providers (incorporated from a parallel workflow) + example and config updates
  • Operational hardcoded values folded into configuration (max_pages/response-body limit/robots TTL + retries)
  • /v1/scrape optionally respects robots.txt + configurable User-Agent
  • S-tier in-house capability absorption, second batch (dbnexus/garrison)
  • Absorbed in-house library capabilities (quick-win batch + medium batch)
  • migrate wreq 5.3 → 6.0.0-rc.31
  • upgrade garrison 0.8 → 0.9.0-rc.1 + bump self-deps to rc.2
  • POST /v1/map site URL discovery endpoint (including making the SSRF switch effective)
  • sitemap parsing and /v1/map business logic
  • /v1/map request DTO and validation
  • Wired up retention batching/timeouts/mutex (cleaned up leftover imports)
  • RetentionWorker multi-instance mutex
  • RetentionWorker per-category timeouts
  • audit_logs batched bounded cleanup
  • webhook_events terminal-state batched bounded cleanup
  • geo_restriction_logs batched bounded cleanup
  • scrape_results batched bounded cleanup
  • retention single-batch short-transaction deletion skeleton
  • RetentionBatchPolicy bounded-deletion parameters
  • retention batching/timeout/mutex configuration section
  • bootstrap admin key switched to 0600 file delivery (aligned with the Jenkins/GitLab pattern)

Bug Fixes

  • audit_service error mapping: RepositoryError::DatabaseError has two states (platform rebuilds DbErr::Custom, the lightweight side uses Database(String)), preserving the Database variant semantics
  • i18n locale test environment race condition (CRAWLRS_LANG process-level env serialized via ENV_MUTEX + explicit cleanup)
  • rustdoc 1.98 strict mode: 88 cases (bare links converted to code font/full paths, lazy continuation indentation, bare generic HTML escaping)
  • clippy 1.98: explicit result_large_err exemption, cmp_owned eliminated
  • sticky TOCTOU race-condition test window 1ms→50ms, eliminating intermittent failures under CI load
  • Merge remote-tracking branch 'origin/main': unified the dual-remote baseline (after the rebase, same-named fixes landed on gitee first)
  • Fixed all red legs in CI/Release — deny/matrix/acceptance/sdk/MSRV
  • Fixed all red legs in CI/Release — deny/matrix/acceptance/sdk/MSRV
  • 1.98 toolchain fixes: doc links/HTML escaping/.locale test race condition/result_large_err (88 cases)
  • RepositoryError::DatabaseError two-state mapping preserves the Database variant semantics
  • Added the missing redis 1.7.0 sha1_smol entry to Cargo.lock
  • Enqueue outbound crawl links before the adaptive-stop determination, fixing depth>1 unreachability
  • Dual-layer SSRF protection for outbound webhooks + closed out strix/OCR audit leftovers
  • Removed hardcoded leaked credentials from test_scenarios.py (CWE-798/CWE-540)
  • Unified JSON extraction rejection and authentication errors in the ApiResponse wrapper
  • scrape route assembly injects an Option-shaped Extension, fixing all POST /v1/scrape 500 errors
  • wait_audit_tasks tolerates cancelled tasks, so shutdown waiting no longer panics
  • ShutdownCoordinator::wait_for_completion only waits for the trigger signal, fixing the worker's 30s self-termination
  • Review-issue fix batch + e2e suite hardening + test reinforcement
  • Fixed the garrison_listener flaky test's global-state race condition
  • Added skip guards to handler/middleware DB-dependent tests
  • Added skip_if_no_test_db guards to DB-dependent tests
  • remove needless PoolConfig::default() update
  • Closed out verification issues on the latest main — workers geo gating regression + acceptance/webhook lint (full-matrix re-verification passed)
  • Completed audit/geo policy parameter call sites + acceptance fmt
  • webhook signature unit tests switched to the free function generate_signature (the method was removed with the msg_id fix)
  • fix(webhook)+test(acceptance): msg_id consistent with the webhook-id header (signature-verification defect) + webhook/SDK/extract in-depth acceptance scenarios
  • Completed retention mock signatures under the test-mocks/unit combination + unix-only test platform gating
  • Added a kind=path annotation to the /v1/crawl/{id} path parameter (sdforge infers GET parameters as Query by default)
  • Carried the SSRF disable switch through SsrfValidator::validate and the engine layer
  • e2e scripts changed from hardcoded API keys to environment-variable injection
  • converge reinforcement — API-shape scheduling + configuration lower-bound validation + test assertions
  • testcontainers max_connections 2000 eliminates intermittent parallel pool-creation timeouts (addressing the flaky root cause)
  • Fixed clippy/fmt gates after the stable toolchain upgrade
  • Added the dbnexus dependency to the auth side (the garrison_interface bridge needs DbPool; fixes CI Feature Matrix auth/teams-only compilation)
  • Explicit minimal permissions for the check-tech-debt workflow (CodeQL missing-workflow-permissions)

Other Changes

  • All in-house dependencies switched to the crates.io registry: confers/inklog/dbnexus/sdforge/oxcache/limiteron → 0.x.0-rc.5, trait-kit → 0.5.0-rc.6, garrison → 0.9.0-rc.2, vecboost → 0.3.0-rc.1; removed all ../base and ../garrison cross-repository path dependencies and the [patch.crates-io] section, so CI no longer depends on sibling-repository checkouts
  • Merged the forked histories of the github/main dual remotes (dual-remote documentation alignment, BDD acceptance-line material retained in history)
  • Added the 0.2.0-rc.2 section (Version Consistency gate)
  • unify codeql-action to 4.38.0 (supersedes #36/#37)
  • bump github/codeql-action/init from 4.37.7 to 4.38.0
  • Merge remote-tracking branch 'github/main'
  • All in-house dependencies switched to the crates.io registry; bumped to 0.2.0-rc.2; incorporated parallel in-flight changes
  • Corrected the test-matrix database description from "only supported backend" to three-backend support
  • Corrected the architecture documentation's database-backend description from PostgreSQL-only to three-backend support (PostgreSQL/SQLite/MySQL)
  • Cleaned up trailing whitespace and end-of-file newlines
  • Added a typos allowlist to eliminate false positives
  • Integrated the detect-secrets baseline
  • Integrated pre-commit gates (formatting/clippy/check/private-key scanning)
  • Cleaned up redundant old copyright headers in test modules and completed CODEOWNERS/fixtures attribution
  • Synced README/FAQ attribution years and di directory copyright headers
  • Completed the copyright line in the LICENSE header (Apache-2.0 dual copyright notice)
  • validator converged to the sdforge::validator re-export (incorporating parallel-session changes)
  • sea_orm integrated via the dbnexus re-export (incorporating parallel-session changes)
  • cargo fmt (formatting fixes required by the agentstem workspace hook)
  • Existing maintenance changes from parallel sessions (CI/documentation/dependencies, etc., outside the i18n scope)
  • Corrected the description of Bloom gating in the interner comparison table (this crate has no bloom feature)
  • Incorporated comment-cleanup changes (before merging back to main)
  • include_patterns seed switched to en.wikinews.org + added a tag-stripping extraction case
  • Completed the CrawlConfigDto.max_pages test-fixture literal
  • Finished migrating engines traits shim references (unit-test imports corrected to canonical paths)
  • Removed the backward-compatibility layer, unified on the latest code paths
  • include_patterns seed switched to en.wikinews.org + added a tag-stripping extraction case
  • E2E/integration tests all switched to real news sites, example.com prohibited
  • Recorded verification of leaked-key invalidation and closeout of outstanding items
  • Ignored the api_test.sh smoke-test output artifact test_results.txt
  • Incorporated in-flight R-key annotation cleanup (19 files, comment-only changes)
  • Test-quality hardening report (14 defect fixes + coverage matrix + audit notes)
  • Semantic-level API E2E suite and live orchestration (Stage 6 liveapi)
  • Fixed LimiteronService signature drift and serialized shared-library queue semantics tests
  • README hero v2 redesign (removed redundant spacing/eliminated tagline duplication/fixed an orphan character in the subtitle)
  • Documentation system updated
  • Fully synced dependencies to the latest and normalized minor-level version notation
  • Removed the unused webhook_repo field from CrawlUseCase
  • Removed BOOTSTRAP-DIAG debug code
  • Removed the outdated tls_fingerprint_check example
  • Unified CapturingLogger as a shared test_utils module
  • Removed the regex_cache dead-code propagation chain + fixed allow(dead_code)
  • align local crate version constraints with rc.1 base crates
  • Unified the offline filtering mechanism (sdk dual tagging) + retries guard + CI aligned with env offline mode
  • Added a scheduled cleanup workflow to delete leftover dependabot/* branches
  • cargo fmt (webhook_service signature block formatting)
  • Acceptance coverage updated
  • Added the map unauthenticated 401 scenario + linked the benchmark comparison in EN README (converge)
  • docs+ci(benchmark/guide/acceptance): linked the comparison report, acceptance usage instructions, CI acceptance job
  • Full BDD acceptance-scenario coverage (8 features, 40 scenarios) + made the SSRF switch effective + sdforge path annotation fix
  • Ignored the .workbuddy-ai/ workspace directory
  • Cucumber scenarios and wiremock support layer for the six major business endpoints
  • cargo fmt (map test assertion line b...
Read more

Release v0.2.0-rc.1

Release v0.2.0-rc.1 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 25 Aug 21:25

This release upgrades dependencies github/codeql-action/analyze, github/codeql-action/autobuild, github/codeql-action/init, etc., adds scrape cache TTL decoupled from search and a retention cleanup scan index, fixes the sticky TOCTOU race-condition test TTL 1ms→50ms… and the rate_limit_helper result-large-err allow…, and adds the library crate not tracking Cargo.lock… and the CHANGELOG 0.2.0-rc.1 section (release gate).

Features

  • web-axum feature: the auth-off web middleware build surface (platform implicitly enabled auth, so this path was not buildable in the original feature matrix; it can now be built/tested independently)
  • allow_unauthenticated_protected() explicit opt-in (auth-off surface)
  • scrape cache TTL decoupled from search
  • retention cleanup scan index
  • Registered RetentionWorker and injected the cleanup data source
  • RetentionWorker unified retention-period cleanup
  • webhook_events terminal-state event cleanup interface
  • geo_restriction_logs retention-period cleanup interface
  • scrape_results retention-period cleanup interface
  • Added a retention retention-period configuration section
  • All in-house dependencies switched to local paths (dbnexus/inklog/sdforge/limiteron/garrison/trait-kit/oxcache/confers)
  • add max_results config to cap API limit
  • add bootstrap service for initial admin setup
  • rewrite SmartSearchEngine as concurrent aggregator with RRF fusion
  • add dedup and rrf modules for smart engine refactor
  • add Exa/Parallel/Tavily fallback search engines
  • remove Google search engine
  • add locale/country support to all search engines
  • Added a readiness probe and an in-process event bus
  • T093 integrated KG structural-hole signals into the frontier scoring pipeline
  • T059 MLLM autonomous-navigation crawl example
  • T072-T087 frontier algorithm exploration
  • T061-T071 Prometheus metric instrumentation + test coverage + benchmark extension
  • T045-T060 MLLM autonomous-navigation crawl engine
  • Phase 1 TLS fingerprint-spoofing engine — wreq BoringSSL backend/JA4 fingerprint
  • Phase 0 security hardening — shutdown timeout fix/SSRF default deny/startup security checks
  • Added task-management API examples (task query, batch cancel, synchronous wait)
  • Completed rate-limiting examples (basic rate limiting, circuit breaker, team concurrency control)
  • Completed proxy examples (HTTP proxy, proxy authentication, proxy rotation)
  • Completed extract examples (multi-rule extraction, output formats, XPath extraction)
  • Completed cache examples (cache configuration, oxcache integration, TTL management)
  • Completed auth examples (API Key, Bearer Token, scope validation, team isolation)
  • Completed advanced examples (async batch, streaming, error handling, metrics export)
  • add extraction_prompt and extraction_schema support
  • only_main_content body extraction integration (T074/R-content-001)
  • T028 retry classification + T067 adaptive stop condition
  • standardwebhooks signatures + hickory-resolver DNS
  • Implemented a pgmq-style message queue (repository pattern)
  • integrate locale-aware messages in handlers
  • localize error messages and response builder
  • add Accept-Language middleware and route registration
  • Added the Mozilla Fluent i18n system
  • Configuration/i18n/infrastructure improvements
  • Stage 4 WaitFor + Hedge integration (T069-T070)
  • T068 TabPool reuse layer (DashMap + CAS LIFO stack)
  • T066 should_crawl delegates to FilterChain + extract_and_queue_links uses Frontier
  • T065 Frontier priority queue + domain round-robin
  • T064 URL scorer trait + CompositeScorer + keyword/path-depth scorer
  • T063 URL filter trait + FilterChain + three concrete filters
  • Stage 3 closeout T057-T062 advanced cache modes + MRT waterfall timeouts
  • Stage 3 P0 proxy rotation pool and ProxyProvider abstraction (T054-T056)
  • Stage 3 P0 body extraction and URL layered dedup (T044-T053)
  • Stage 2 scheduling infrastructure and identity rotation (T020-T042)
  • Stage 1 P0 core protections and scheduling (T008-T019)
  • Stage 5-7 + T034 full verification + pre-commit review fixes
  • T030 — migration 005 marks the old api_keys/scopes tables as deprecated
  • Stage 6 — API Key issuance + old-table deprecation + re-signing script (T027-T029)
  • Stage 5 — CrawlrsAuditListener bridges garrison events to the crawlrs audit service (T024-T026)
  • Stage 4 — removed hand-written authentication dead code + security/performance fixes (T018-T023)
  • Stage 3 — middleware bridging and garrison RBAC rewrite (T012-T017)
  • Stage 2 — DI initialization and BootstrapError::Auth (T010-T011)
  • Stage 1 — garrison infrastructure setup (Config/DAO/Interface)
  • Stage 0 — prerequisite gate verification + garrison dependency introduced (T001-T003)
  • Stage 4+5 — webhook gating + feature matrix/CI/docs acceptance (T023-T035)
  • Stage 3 — rate-limit feature gating and NoopRateLimitingService (T017-T022)
  • Stage 1+2 — auth default-identity and teams single-tenant degradation (T006-T016)
  • Stage 0 — feature scaffolding and default-identity constants (T001-T005)

Bug Fixes

  • exe_path cross-platform naming (incorrect _exe_path reference on the Linux side)
  • Missing test construction for three new WorkerManagerDeps fields (added minimal test doubles)
  • clippy -D warnings:result-large-err / struct update no-op / unused variable
  • rustdoc link debt (crate-level lint declaration)
  • cargo-deny: lru upgraded to 0.18.2; paste transitive-dependency exemption recorded (RUSTSEC-2024-0436/2026-0258)
  • sticky TOCTOU race-condition test TTL 1ms→50ms (the in-lock reselect window was too narrow, causing threads to reselect different URLs after expiry)
  • Added the missing rate_limit_helper result-large-err allow
  • Gate closeout — three clippy -D warnings items (result-large-err/struct update noop/exe_path unused), deny exemption for paste, lru upgrade to 0.18.2
  • wreq declared as a 0.16 placeholder (upstream 5.x all yanked/6.0-rc broken, resolve gate); adaptation recorded
  • cargo-deny with explicit --features full (isolating the wreq 5.x upstream all-yanked incident) + rustfmt normalization
  • Gate fixes — lib cross-platform exe_path naming, rustdoc link-debt lint declaration, WorkerManagerDeps three-field test doubles and cleanup_expired completion
  • Added --allow-dirty to cargo publish (so Cargo.lock resolution updates do not block the release)
  • workspace.package version synced to 0.2.0-rc.1
  • fix(auth-off)+test(purify): protected routes default to 401 + anonymous restricted-identity opt-in (T037/T038), Noop made explicit (T039), sdk integration-layer duplicate cases purified (T041), web-axum feature unlocks the auth-off build surface
  • Local path dependencies unified to relative paths (cross-platform fix)
  • dbnexus-macros local path (eliminating the proc-macro-error2 warning pulled in by registry 0.4.4)
  • Eliminated unused_mut/unread-assignment warnings (final_content lazily initialized as immutable)
  • Route registration gated by the http feature + chromiumoxide 0.9.1 API adaptation + ssrf migration test fixes
  • resolve 30 bugs from OCR deep scan report
  • align CacheWriter::set signature with oxcache 0.4 trait
  • resolve compiler warnings in webhook and wreq engine
  • fix compilation errors from module renames and API changes
  • correct trigger function name in message queue schema
  • use concrete repo impls for v2 Extension type matching
  • replace SSRF host rewrite with resolve override
  • inject FlareSolverr timeout from config, unify fallback URL
  • remove weak password defaults from docker and test configs
  • Fixed the crawl example's CrawlConfigDto missing extraction_prompt/extraction_schema fields
  • allow dead_code lint for sdforge forge macro
  • Fixed the proxy_pool sticky TOCTOU race condition + worker timing flaky test
  • Fixed pre-existing test compilation errors (import paths, trait signatures, type mismatches)
  • Removed the invalid exclude field + cleaned up configuration
  • handlers/middleware fixes and optimizations
  • Fixed tests/main.rs test-target compilation errors and garrison singleton contention
  • Fixed HIGH/MEDIUM issues found in the subagent review
  • Merge dev/v0.2.0: three-dimensional review HIGH/MEDIUM fixes + documentation sync
  • Three-dimensional review HIGH/MEDIUM fixes + documentation sync
  • T067 registered the adaptive module + hedge reset warning fix
  • T001+T002 security configuration hardening
  • Fixed the SSRF IPv4-compatible IPv6 bypass vulnerability and strengthened test coverage
  • Explicitly specified the tool parameter for cargo-llvm-cov
  • Added a User-Agent header to crates.io API requests to avoid 403

Other Changes

  • auth-off security semantics tightened: protected routes deny by default (401), removed the DEFAULT_IDENTITY_TOKEN_HASH placeholder hash and full_access() identity injection; after opt-in, an anonymous restricted identity is injected (denied() scope, no token_hash)
  • NoopRateLimitingService/NoopWebhookService assembly logs raised to one-time error level (making the security-semantics degradation explicit)
  • sdk route tests converged to the unit layer (removed semantically duplicate cases between tests/sdk_api_test.rs and src/presentation/sdk/tests.rs)
  • wreq dependency declaration changed to a 0.16 placeholder (upstream incident: 5.x all yanked, 6.0.0-rc broken builds; enabling engine-tls-fingerprint requires adaptation to the 0.16 API)
  • protected routes default to 401 under auth-off (embedders that relied on all-open unauthenticated access must opt in explicitly)
  • Library crate does not track Cargo.lock (CI publish resolution refresh makes it dirty)
  • CHANGELOG 0.2.0-rc.1 section (release gate)
  • Reverted --allow-dirty (no bypassing the dirty gate)
  • Cross-repository dependencies made version-only + local [patch.crates-io] redirection (CI release mode)
  • bumped crawlrs to an rc version (crates.io +0.1 minor prerelease)
  • auth-off default 401 + opt-in new semantics
  • Eliminated clippy warnings (field_reassign_with_default/redundant_pattern_matching/unused import, etc.)
  • DbPool test const...
Read more

Release v0.1.0

Choose a tag to compare

@github-actions github-actions released this 21 Jul 22:58

This is the repository's first release; there is no earlier release to compare against. The full commit history is at: https://github.com/Kirky-X/crawlrs/commits/v0.1.0.

Features

  • Web platform and API: added handlers and improved middleware in the presentation layer (c682883), unified the API path prefix (aedb264), generated OpenAPI documentation with utoipa (a84ebf6), and unified the API response format and added a timestamp field (330a837)
  • Multi-engine crawling: reqwest engine (7f41463), conditional compilation of the Playwright engine (6d0b7db), FlareSolverr engine supporting smart routing to bypass CAPTCHAs (8699679); EngineClient unified public API (d7efb71) and ScrapeOptions advanced capabilities (14fffda); HTTP proxy support for all engines (53633b8); automatic browser download (1de2c48)
  • Smart search: unified search module with security hardening (42d5b05), SmartSearchEngine router and factory (9f7bca4), rate limiting/timeout control and test-data loading (531596d)
  • Tasks and queues: task state machine and dedup optimization (7ee4008), QueueClient unified queue interface (31f2f8f), PostgresTaskQueue (a1cf7a1), automatic task expiry and access control (1fe2446), improved retry strategy (8d38ec1)
  • Access control and teams: API key scope management, feature flags, audit logs (6f17710), persistent team-level domain blacklist (720b413), geographic access restrictions wired to the database repository (deea68d), geographic-restriction data table (2d4443c), improved IP access control and multicast IP blocking (e21335a)
  • LLM extraction: multi-provider support and extraction enhancements for the LLM service (f86f543), prompts.toml prompt configuration (3d88e36), local Ollama model configuration (c276eda)
  • Storage and cache: storage abstraction layer and multiple storage implementations (fee3d75), S3 object-storage integration (7c46dfb), SQLite support and a performance benchmark suite (29a7d04)
  • Underlying library unification: confers fully took over configuration (287b47b), dbnexus fully refactored database operations (7149827), limiteron refactored rate limiting/concurrency control/quota management (821ba45), inklog fully replaced tracing (19996de), sdforge interface wrapping (999c8e4), CacheService trait and OxcacheService (af628bb)
  • Other core capabilities: text-encoding processing module (24c0d08), smart port-sniffing module with default port 8899 (d09e3a7), degradation strategies and alert rules (b29fc62), completed WebhookManagement/TeamManagement/ScrapeService domain services (361bf80), webhook management and worker (6b5ad3c), environment-variable validation and infrastructure modules (988fd79), completed database/config/sdk examples (2ed1998)
  • Engineering infrastructure: criterion benchmarks (a5d1900), Docker test environment (004d55b), feature test matrix (9bea9b9), Python API test framework (a667f6e), complete CI/CD harness configuration (47b5a94)
  • Fixed 3 strix vulnerabilities + configured the complete CI/CD harness
  • Completed database/config/sdk examples (T024-T026)
  • T049 follow-up — removed the testcontainers redis dependency
  • T049+T047c+T044 — completely removed the Redis runtime dependency
  • T047+T048 — migrated RobotsChecker to CacheService and ScrapeWorker to TeamSemaphore+DashMap
  • Defined the CacheService trait and implemented OxcacheService (T045+T046)
  • Updated bootstrap module initialization to trait-kit (T040)
  • AppState built via AsyncKit, removed state_manager (T039)
  • Implemented trait-kit module definitions and AsyncAutoBuilder (T037+T038)
  • T020-T022 completed domain services — ScrapeService + WebhookManagement + TeamManagement
  • T016-T019 queue industrialization — MessageQueue trait + InMemoryQueue + ApalisQueue + adapters
  • Phase 7 environment-configuration improvements — CI/Docker/env (T022-T024)
  • Phase 6 feature-gating improvements — gated optional dependencies + cfg guards (T020-T021)
  • Phase 3 sdforge interface wrapping (T014-T016)
  • inklog fully replaces tracing (Phase 2: T009-T013)
  • Migrated local path dependencies to the latest crates.io versions
  • Added the as_inner method and a database session wrapper
  • update create_pool and add Default impl for DatabasePool
  • add payload limit extraction for tasks
  • add crawl use case tests
  • merge test enhancements from worktree
  • Added rate-limiting service and task state-machine test files
  • add task queue tests
  • Optimized the code structure
  • Continued handling user requests
  • Refactored rate limiting, concurrency control, and quota management using the limiteron library
  • Fully refactored database operations to use the dbnexus library
  • Added inline default-value support
  • Used the confers library to fully take over configuration management
  • complete project refactoring with confers config migration
  • Added a timestamp field to the unified API response format
  • Added automatic browser download
  • Optimized dependency imports and reduced binary size
  • Added conditional-compilation guards for the Playwright engine
  • Merge branch 'feature/modernization-2026-01'
  • Phase 3 DTO, use-case modules, and domain-event mechanism
  • Phase 1-2 infrastructure and API standardization
  • add utoipa for OpenAPI documentation
  • Added new infrastructure components
  • Enhanced the LLM extraction example
  • add configuration service and documentation
  • improve configuration structure and validation
  • Added authentication rate limiting and DTO validation
  • Added a pre-commit check script
  • Split error types into a standalone module to improve code organization
  • Unified constant management and added 15+ key constants
  • update all Rust dependencies to latest versions
  • add environment variable validation and infrastructure modules
  • migrate CreateScrapeUseCase from EngineRouter to EngineClient API
  • Configuration externalization and error-handling improvements
  • add SafeUrl type and validated_url field to DTOs
  • add code review report and secret handling utilities
  • add dependency injection module (shaku)
  • update engine router demo with enhanced test scenarios
  • enhance LLM service with multi-provider support and improved extraction
  • enhance LLM extraction test with markdown and JSON formats
  • add prompts.toml and update LLM settings
  • Moved engine configuration from hardcoded values into configuration-file management
  • Added FlareSolverr engine support implementing smart routing to bypass CAPTCHAs
  • Fixed the Sogou search parser's URL-parsing issue
  • Add .sisyphus directory to .gitignore
  • Add AGENTS.md files to .gitignore
  • Add .claude and .shared to .gitignore
  • Added project documentation and a logo
  • add new handlers and improve middleware
  • add HTTP proxy support for all scraping engines
  • Completed a comprehensive production-environment test plan
  • Added test-runner scripts and documentation
  • Added the feature test-matrix configuration
  • Added a complete API test suite
  • Added a Python API test framework
  • Added a complete Docker test-environment configuration
  • Added a Docker test environment and headless Chrome support
  • optimize critical path performance
  • add API key scope management, feature flags, and audit logging
  • Extend ScrapeOptions with advanced capabilities
  • Add engine critical issues fix proposal
  • Implement EngineClient as unified public API
  • refactor config module into sub-modules
  • add unified QueueClient with proper validation
  • add unified search module with safety hardening
  • Add project documentation and test utilities
  • Add test data support to Sogou search engine and enable search engine tests
  • Enhanced HTTP handler functionality and error handling
  • Improved application startup and worker management
  • Enhanced core functionality of task and crawl services
  • Enhanced storage service functionality and error handling
  • Improved the robustness and configurability of the rate-limiting service
  • Added the crawlrs API and worker services to docker-compose.test.yml
  • Added benchmarks for core system functionality
  • add Google search integrationSolverr support
  • add browser automation tests with remote Chrome support
  • add max_attempts parameter to control port scan range
  • add Google search engine with FlareSolverr to test suite
  • add rate limiting, timeout control, and test data loading to SmartSearchEngine
  • enhance Google and Sogou engines with logging improvements
  • improve logging format and assertions
  • enhance with rate limiting and anti-crawl measures
  • add engine registration and listing methods
  • add SearchResult Default implementation and update plan
  • add smart search engine router and factory
  • include Bing, Baidu, Sogou in test search aggregator
  • Completed implementation of degradation strategies and alert rules
  • Implemented automatic task expiry and access control
  • implement persistent team-level domain blacklist
  • integrate database-based geo restriction repository
  • implement S3 object storage integration
  • update task entity and repository implementation
  • enhance crawl and scrape core logic
  • update webhook management and worker
  • update search engines and router
  • implement team management and geographic restrictions
  • add geographic restrictions table
  • Integrated leading-newline handling into the web content handler
  • Added leading-newline removal for strings
  • enable only Sogou search engine for "星光盛典" search
  • Configured the Google search engine and disabled the others
  • Added a text-encoding processing module supporting Unicode detection/conversion, encoding-format detection, encoding conversion, error handling, and performance optimization
  • Update task repository implementation and add expires_at migration
  • Complete boundary scenario validation (UAT-007 to UAT-012)
  • Added a storage abstraction layer and multiple storage implementations
  • Added SQLite support and a performance benchmark sui...
Read more