Repository navigation
SoloCrawl 1.2.0 hardens fetching, proxy recovery and URL safety after an independent code review.
Changes
- Recover failed proxy endpoints after a 60 second cooldown instead of disabling them for the life
of the process; a 429/503 from the target site no longer marks the proxy as failed. - Block shared and other non-global addresses, including CGNAT/Tailscale
100.64.0.0/10. - Give up instead of waiting when
Retry-Afteris longer than the request timeout. - Use the Playwright fallback only for HTML responses (not short JSON or PDF), and return the
rendered page when the browser is forced even if its content is short. - Load robots.txt for different hosts concurrently instead of one at a time.
- Avoid repeated DNS checks for the same URL during a fetch.
- Add
select_providers(config, sources=...)in the search core; the MCP and CLI shells use it. - Log the exception type when a browser fetch fails.
- Remove dead code (legacy
duckduckgo_searchimport, signal-handler cleanup that never ran) and
the stalerequirements.txt.
Compatibility notes
ProxyEndpointno longer has ahealthyfield; failure state is tracked byProxyPool.- The legacy
duckduckgo_searchpackage is no longer used as a fallback;ddgsis required (as
already declared in the dependencies). - Install from
pyproject.toml;requirements.txthas been removed. - The five public MCP tool names and signatures remain unchanged.
Verification
The full verification runner passed: Ruff, formatting, Pyright, offline fixture tests (447), MCP
protocol, local Chromium, dependency consistency, vulnerability audit, wheel/sdist build and clean
base/browser/all installations. All ten package registries, the public search providers, and live
federation, fetch, research and batch passed. SearXNG was skipped because no instance was
configured.