mini mo

A local, hardware-light RAG + Tier-0 distillation assistant.

• Read documents (.pdf, .txt, .md)
• Ask questions answered only from your documents (retrieval-augmented)
• Distill a document: a teacher LLM turns it into clean Q&A knowledge that is stored back into the vector DB so future answers get sharper

Everything runs locally. The only thing that touches the network is the LLM provider if you choose the Claude API instead of local Ollama.

Architecture

file (pdf/txt/md)
  └─ ingest.py    extract text → chunk
       └─ embeddings.py  (sentence-transformers, local, CPU)
            └─ store.py  Chroma vector DB  (data/chroma)  ── kind="source"
ask ─ rag.py  retrieve top-k → llm.py (Ollama or Claude) → grounded answer
distill ─ distill.py  teacher reads source chunks → Q&A pairs ─┐
                                                               └─ store.py  kind="distilled"

jobs.py runs distillation in a background thread so the button returns immediately and the UI polls for progress. This is the seam where a real queue + GPU fine-tuning (Tier 1) would plug in later.

Setup

Already done in this workspace: a .venv with dependencies installed.

Choose your LLM engine in .env:

• Local (default): install Ollama, then:

  ollama pull llama3.1:8b
  

• Claude API (best teacher quality): set LLM_PROVIDER=anthropic and ANTHROPIC_API_KEY=... in .env.

Run

./run.ps1

Open http://127.0.0.1:8000 — then: Ingest a file → Distill it → Ask questions.

Where to go next (Tier 1)

When you have an NVIDIA GPU, add a second job kind in jobs.py that runs a QLoRA fine-tune on the distilled Q&A pairs, producing a small adapter you can load in Ollama. The distilled DB you build now becomes that training set.