Skip to content

SoloCrawl 1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@hlavacm hlavacm released this 27 Jun 13:56
v1.0.0
a279b4c

Initial public release of SoloCrawl.

SoloCrawl is a local MCP/CLI tool for personal developer workflows with local AI coding tools such as LM Studio, OpenCode, Claude Desktop, and similar clients.

It provides:

  • federated web search
  • page scraping into clean Markdown for LLM context
  • one-shot research with cited Markdown output
  • live package-version lookup from official registries
  • an MCP stdio server for local AI clients
  • a CLI for direct terminal use

SoloCrawl runs locally, requires no cloud account or API keys by default, and is safe by default for local use. It is not intended to be exposed as a public network service.

Install

Recommended with pipx:

pipx install git+https://github.com/hlavacm/solocrawl.git

Or from a local clone:

git clone https://github.com/hlavacm/solocrawl.git
cd solocrawl
pipx install .

Quick start

solocrawl search "python asyncio semaphore" --limit 5
solocrawl scrape https://example.com
solocrawl package requests --ecosystem pypi
solocrawl research "python asyncio semaphore" --depth 3

Start the MCP server:

solocrawl-mcp

Safety defaults

  • blocks localhost, private, link-local, reserved, and cloud-metadata targets by default
  • checks literal hosts, DNS-resolved addresses, HTTP redirects, and Playwright final URLs
  • honors robots.txt by default
  • caps fetched response size
  • keeps MCP-facing search/research/output bounded

Requirements

  • Python 3.14+

See the README for installation, quick start, and MCP client setup.

Assets 2
Loading