lilbee: a single-executable local search engine that crawls the web into a private library you can talk to, with crawl4ai

[tobocop2]

Thanks for crawl4ai. It's how lilbee turns websites into searchable text. lilbee is a local-first search engine you can talk to: it runs and manages its own local models, indexes your files and code, crawls the web you point it at, and answers with a citation to the source, all in one executable. Point lilbee at a docs site, a wiki, or a forum, and crawl4ai fetches the pages, converts them to markdown, and lilbee adds them to your library. From then on you can search or chat with that copy offline, even after it changes or goes down. It ships as the [crawler] extra.

My earliest use case for lilbee was talking to horticulture forums that I crawled and indexed, and I was blown away by how useful my own hardware suddenly became. Some cool details: recursive crawls emit live progress to the Task Center, respect per-domain rate limits, and retry on HTTP 429/503 with jitter. lilbee sets a gentler default than crawl4ai's own (3 concurrent requests instead of 5) and passes its randomized base-delay range straight into crawl4ai's RateLimiter. SSRF protection blocks internal networks by default.

On the roadmap: lilbee runs inference in-process today and is moving to a llama-server and llama-swap setup so one binary scales across multiple GPUs (tobocop2/lilbee#267).

Site: https://lilbee.sh.
Repo: https://github.com/tobocop2/lilbee