Skip to content

SoloCrawl 1.2.0

Latest

Choose a tag to compare

@hlavacm hlavacm released this 19 Oct 08:36
· 1 commit to main since this release

SoloCrawl 1.2.0 hardens fetching, proxy recovery and URL safety after an independent code review.

Changes

  • Recover failed proxy endpoints after a 60 second cooldown instead of disabling them for the life
    of the process; a 429/503 from the target site no longer marks the proxy as failed.
  • Block shared and other non-global addresses, including CGNAT/Tailscale 100.64.0.0/10.
  • Give up instead of waiting when Retry-After is longer than the request timeout.
  • Use the Playwright fallback only for HTML responses (not short JSON or PDF), and return the
    rendered page when the browser is forced even if its content is short.
  • Load robots.txt for different hosts concurrently instead of one at a time.
  • Avoid repeated DNS checks for the same URL during a fetch.
  • Add select_providers(config, sources=...) in the search core; the MCP and CLI shells use it.
  • Log the exception type when a browser fetch fails.
  • Remove dead code (legacy duckduckgo_search import, signal-handler cleanup that never ran) and
    the stale requirements.txt.

Compatibility notes

  • ProxyEndpoint no longer has a healthy field; failure state is tracked by ProxyPool.
  • The legacy duckduckgo_search package is no longer used as a fallback; ddgs is required (as
    already declared in the dependencies).
  • Install from pyproject.toml; requirements.txt has been removed.
  • The five public MCP tool names and signatures remain unchanged.

Verification

The full verification runner passed: Ruff, formatting, Pyright, offline fixture tests (447), MCP
protocol, local Chromium, dependency consistency, vulnerability audit, wheel/sdist build and clean
base/browser/all installations. All ten package registries, the public search providers, and live
federation, fetch, research and batch passed. SearXNG was skipped because no instance was
configured.