Devlog #0: Hello Insight

[monneyboi]

Hey everyone! New project, new devlog series. Buckle up.

Insight is an AI research assistant for documents. Drop in your PDFs, ask questions, get answers with citations. Think Claude Code, but pointed at your investigation folder instead of a codebase. It runs on your machine — chat model, search, storage — and when you share a collection with a colleague, your laptops talk to each other directly. No server in the middle.

Two weeks of work, ~160 commits, and somehow it already does the thing. Let me walk you through it.

The stack

We picked things that let us ship one binary and have it work offline:

• Tauri 2 for the shell — Rust backend, web frontend, no Electron, no Node runtime shipped to your machine.
• Svelte 5 for the UI. Fast, small, no ceremony.
• mistralrs for inference. It loads GGUF models from HuggingFace and runs them on CPU, CUDA, or Metal. We use it for the chat model and for embeddings, which means one inference engine, one set of GPU builds, one thing to keep healthy.
• iroh for everything peer-to-peer. iroh-blobs for content, iroh-docs for metadata, iroh-gossip for live updates. Modern QUIC underneath, handles NAT traversal so you don't have to.
• milli (the library that powers Meilisearch) for hybrid full-text + vector search. Lives in-process, no separate search service.
• lopdf for PDF text extraction.

You can pick a model from the UI — Qwen3 8B Q4_K_M (default), Qwen3 4B for smaller machines, or Qwen3 8B Q8_0 for higher quality — or switch the chat to OpenAI or Anthropic if you'd rather. Your documents stay on your machine either way.

The honest state of "local"

Worth being upfront about this. I'm running Qwen3 8B Q4_K_M on a GeForce 4070 and it works — context is tight, but it's usable. On a laptop CPU you'd be waiting forever for every token, and the experience falls apart. So today, "local" really means "on a decent GPU", and cloud models are there as the pragmatic path for everyone else.

The bet is that in a couple of years, a normal consumer machine will run a useful 8B model comfortably. Hardware is moving in that direction, quantization keeps getting better, and small models keep getting more capable. We want the architecture ready for that day, not to be rebuilt for it. That's the north star — local today where it can be, local-by-default when the hardware catches up.

Why peer-to-peer

It would've been easier to run a server. We didn't, on purpose.

Centralized infrastructure doesn't mix well with journalism. A hosted service can be subpoenaed, blocked, quietly shut off, or simply disappear when the company behind it does. Your investigation should not depend on any of those things not happening. If every peer in a team holds a full copy of a collection — documents, text, embeddings — the work survives any one machine going dark, and it survives us going dark too. No internet? You can still read your files and ask questions. This is basic stuff for a journalism tool, and it's much easier to design in from day one than to bolt on later.

The quieter reason is where it gets interesting: peer-to-peer lets the heavy processing move, not just the data. Two things stay on every peer's machine no matter what — the search index and the agent. Both are inherently local: the index is built per-machine from the synced data, and the agent runs wherever you typed your question. Everything else is fair game. Today every machine also does its own text extraction and embedding generation, but those are one-shot batch jobs whose outputs (text, embeddings) are exactly what already syncs between peers. The natural extension: if one machine in a team has a GPU, it could carry the extraction and embedding for the whole collection, and the precomputed work flows to everyone else through the same channel. None of that is built yet, but the topology it needs already is.

Mechanically: a collection is an iroh-docs namespace, sharing creates a ticket, documents and embeddings both sync automatically. If your collaborator already has the same embedder configured, they don't redo the expensive computation.

A simpler pipeline

The first version of the import pipeline was the obvious one: a job queue, workers, a coordinator tracking which document was at which stage. It worked. It also had all the usual coordinator problems — drift between "what the coordinator thinks" and "what's actually on disk", retries that need bookkeeping, and two separate code paths for "I just imported this locally" vs. "this just arrived from a peer."

Then we noticed: iroh-docs already emits events whenever an entry is inserted. We were writing to it anyway. Why is there a coordinator at all? Why not let the events be the pipeline?

So we ripped the coordinator out:

LOCAL IMPORT                       REMOTE SYNC
────────────                       ───────────
store_pdf_source()                 Peer sends documents
      │                                  │
      ▼                                  ▼
InsertLocal(source)              InsertRemote(text)
      │                                  │
      └──────────► WATCHER ◄─────────────┘
                      │
     ┌────────────────┼────────────────┐
     ▼                ▼                ▼
 files/*/source   files/*/text   files/*/embeddings/*
     │                │                │
     ▼                ▼                ▼
  Extract           Embed             Index

A watcher subscribes to iroh-docs events per collection. Extract workers wait for files/*/source writes, pull the PDF, extract text with lopdf, write it back. The new files/*/text write fires an event that wakes the embed workers. Their output fires an event that wakes the index worker. Local imports and remote syncs go through the exact same path, because both end up as iroh-docs inserts. There is no separate "import flow."

The refactor came out much lighter than the code it replaced. Usually a sign we were overthinking it the first time.

The agent

The agent has four tools and that's it for now:

• search — hybrid retrieval, 60% semantic / 40% keyword, scoped to whatever collections you've selected
• read_chunk — pull a specific chunk by document ID for follow-up context
• list_documents — what's actually in scope
• get_collection_terms — top terms by document frequency, so the model can orient itself before searching blindly

There's no user-facing search bar, on purpose. The agent is the interface. You ask a question in plain language, it searches, reads, and answers with citations pointing at document names and pages. Short answers by default. It goes deeper if you ask.

Modest tool set, but it works end-to-end, which was the goal for this stretch.

What's next

Plenty. Foundation is down, surface is thin:

• More than PDFs. Investigations come as everything — emails, spreadsheets, scanned images. We need to widen the intake, and that probably means a real OCR step.
• Retrieval quality. Hybrid search works in the lab. Real journalism queries on real collections will absolutely break it in interesting ways. We need to actually sit down with one and watch it fall over.
• Citations that go somewhere. Right now a citation is text. It should be a click that opens the exact page in the source PDF. That's plumbing, but it's the plumbing that makes you trust the thing.
• Sync UX. P2P sync works, but the experience of "who has what, what's pending, what's out of date" barely exists yet.
• Sharing processing across peers. The future I teased above — letting one machine in a team handle extraction and embedding for the whole collection, with precomputed results flowing to everyone else. Bigger undertaking than it sounds, and the thing that makes the P2P design actually pay off.

So that's where we are. If you've built anything in this space, or done real investigative work with tools like this, I'd love to hear from you. The shape of the right tool here isn't obvious yet, and the people doing the work usually know it better than the people building it.

More soon 👋