lilbee: the whole local AI stack, packaged in one executable

[tobocop2]

I've been working on lilbee (GitHub) and its companion plugin obsidian-lilbee (GitHub) for many months now, and I finally think it's worth sharing here.

Ask a question and get a cited answer back, entirely from local models. This one is against an indexed car manual, with the GPU visible doing the work:

I have been using lilbee for a lot of programming tasks, but also Q&A tasks, like talking to my car manual and horticulture forums. I have had so much fun working on this and using it.

Why I built it

On its own, local AI is a fun novelty. What makes it genuinely useful is pairing it with an extraction, retrieval, and search pipeline, and that's the crux of lilbee.

I've tethered the inference engine to extraction and retrieval, so now I have a genuinely powerful way to run AI locally. It's all packaged into a single executable: one command and it's there.

You install it with your favorite package manager, from source, or from my GitHub releases. No container orchestration, no spiderweb of dependencies to setup up yourself. That part is owed entirely to Nuitka.

The TUI is just lilbee. Pairing it with an agent (opencode and hermes at the moment) is lilbee launch opencode or lilbee launch hermes. I predominately use opencode for dev work, and I'm open to adding other agents like Claude Code if the interest is there.

Getting started

Model downloads are integrated directly with Hugging Face using the huggingface_hub API client, so the steps to get going are as simple as I could conceivably make them: install one thing, browse, pull a model, ask a question.

In that one binary, you get a model browser that searches Hugging Face Hub, a model manager with multi-GPU placement, an extraction pipeline (90+ document formats plus OCR), a retrieval pipeline, a web crawler, a generated wiki, a chat TUI, an MCP server and agent launcher, a REST API, and the Obsidian plugin.

And it runs on whatever hardware you have, across macOS, Linux, and Windows:

• Apple Silicon (Metal)
• NVIDIA (CUDA or Vulkan)
• AMD (ROCm or Vulkan)
• Intel Arc (Vulkan)
• CPU-only works too, including ARM boxes like a Raspberry Pi 4

Getting a model: browse the catalog, search Hugging Face Hub from inside the app, and pull.

And it's fast

Do not be deceived by this being built in Python. The entire hot path is Rust, C, and C++: inference is llama.cpp, all extraction is done by xberg (which I'm a maintainer of), and storage and search sit on LanceDB. The heavy lifting in the retrieval pipeline, the vector math and the bulk movement of data, is delegated to numpy and pyarrow, which are C and C++ under the hood.

Python is primarily the orchestrator, the TUI, and the HTTP surface, none of which are hot paths. I've profiled this: during a full ingest, xberg's extraction runs with the GIL released, in parallel, at about 1% of the serialized Python time. lilbee's own Python is about a megabyte inside an executable that ships hundreds of megabytes of native code. Developing the orchestration entirely in Python is a joy, and the ecosystem around it is what keeps the native core fast.

Web crawling

lilbee can also crawl websites. Point it at a docs site or a wiki, and the pages get fetched, converted to markdown, and indexed, so you can search and chat with your own offline copy locally, even after the site changes or goes down.

This is valuable for agents too, even ones with no crawler built in. Web-tool agents fetch one page at a time; crawl a site into lilbee once and every agent gets the same offline, cited copy over MCP, nothing leaving your machine. They can kick off crawls themselves, and the full set of MCP tools is in the usage guide.

Here it crawls a whole site, following links one level deep, then a single question gets a cited answer drawn from across the pages:

It's not just me anymore

I have completely non-technical, non-dev folks using lilbee, often replacing their need for frontier AI altogether.

One friend recently built a game in Godot 4 entirely with free-tier AI tools: opencode zen with MiniMax M2.7, pointed at lilbee's MCP server with the latest Godot docs indexed. It worked remarkably well.

Another friend uses the lilbee TUI on his laptops paired with some smaller models like Jan v3.5 4B. My favorite use case of his is that he used lilbee to beat Myst completely offline by ingesting the strategy guide for it, and he got further than he ever had in his life. He also used the same method to beat Star Wars: X-Wing.

Another friend that's not a developer, uses lilbee for solar panel research entirely offline: he ingests the documentation and asks it questions. It's thrilling to see non-devs using local AI.

Agents on local models

One model serves as many agents as you want to run. These reels show four working at once against a single local model, each in its own worktree, reading and searching this repository through lilbee.

Qwen3 Coder Next, 45GB across three RTX 4090s: four agents on one model, 138 tokens a second warm.

Gemma 4 26B is a thinking model, and lilbee streams its reasoning as visible text: every pane shows the model working through the problem before it answers.

Qwen3.6 27B on the same four tasks: each agent states what it expects, then reads the code to check itself.

And the same setup scales up: Devstral 2 123B, 70GB across two A100 80GB cards, one agent at the model's full 14 tokens a second.

All of this runs on llama-server as the inference engine, with llama-swap keeping a server per role resident behind one endpoint and gguf-parser estimating each model's memory footprint for placement. I've done very intense development tasks with this combo.

Why not llama.cpp's own fit and router machinery? I'm tracking both: router mode as a llama-swap replacement and fit-params as a gguf-parser replacement. Neither covers what placement needs yet, so I'm reliant on llama-swap and gguf-parser at this time.

Big models, multiple GPUs

When a chat model won't fit on a single GPU, lilbee spreads it across the ones you have. It sizes each model with gguf-parser and picks the split automatically, or you can pin the layout by hand.

Here Llama 3.3 70B, about 40 GB of weights that no single consumer card holds, lands across three RTX 3090s with no configuration at all, and answers a question from an indexed car manual with page citations:

A wiki written from your documents

lilbee also writes a wiki from whatever you've indexed: one page per concept or entity, not per document, cross-linked with [[wiki links]], and every section is citation-verified against the source text before it publishes.

Here I browse a few articles, then search for a page that doesn't exist yet, and lilbee writes it on demand:

Beyond the laptop

lilbee scales well beyond my laptop. I've ingested millions of files across many GPUs, and I made my best effort to grade its search engine, you can check out my benchmark.

Benchmarks aside, talking to those documents locally has been a real success.

In one example, I used Mistral-Large-2411 for Q&A against all the recent UFO docs the government released, 789 documents and 7,288 pages of official records, and it did extremely well, giving me answers I preferred to Opus 4.8 paired with the exact same retrieval pipeline. I'm not a tin foil hat in the slightest, but this was a use case my friends were asking me about, so I did it and it was a lot of fun. Here are the Q&A sessions, judge for yourself:

• Mistral vs Claude
• Just Mistral
• Just Claude

Where it's going

So overall, lilbee brings the whole local AI stack into one tool. It's under active development with much more to come, and I think I've largely hit the local AI democratization I set out for. Excited to keep building.

lilbee is built on llama.cpp, which does all the inference and is the core of the whole thing and I'm wildly grateful to this project.

The code is at github.com/tobocop2/lilbee, and the Obsidian plugin is at github.com/tobocop2/obsidian-lilbee (obsidian.lilbee.sh).

Links

• Project site · GitHub · Install
• Obsidian plugin · plugin GitHub
• Retrieval benchmark (MS MARCO)
• MCP tools in the usage guide