rawrag

A local RAG (Retrieval Augmented Generation) system for querying personal markdown notes using local LLMs. No cloud, no data leaving your machine.

Built from scratch in Python — no heavy abstractions, just the pipeline.

What it does

• Ingests markdown files (Obsidian notes, Telegram exports, anything .md)
• Chunks, embeds, and indexes them in a local Qdrant vector database
• At query time: retrieves → reranks → applies MMR diversity → stuffs neighboring context → streams an answer via a local LLM
• Exposes a web UI and a REST API

Stack

Layer	Tool
Embeddings	bge-m3 (multilingual, via Ollama)
Vector DB	Qdrant
Reranker	BAAI/bge-reranker-v2-m3 (CrossEncoder)
LLM	Ollama (Qwen3 9B default, 4B lite mode)
API	FastAPI + SSE streaming
History	SQLite via aiosqlite

Retrieval pipeline

query → embed → Qdrant ANN (top 20) → dedup → CrossEncoder rerank
      → MMR diversity (λ=0.7, top 5) → fetch neighbor chunks
      → lost-in-middle reorder → stream via Ollama

Setup

Requirements

• Python 3.11+
• Ollama running locally
• Docker (for Qdrant, or run Qdrant manually)

Install dependencies

pip install uv
uv pip install -r requirements.txt

Pull models

ollama pull bge-m3
ollama pull dmtx/qwen3.5-9b-abliterated      # default
ollama pull richardyoung/qwen3-4b-instruct-2507-abliterated  # lite

Start Qdrant

docker run -p 6333:6333 qdrant/qdrant

Configure

Copy .env.example to .env and fill in values:

cp .env.example .env

Relevant settings:

GENERATION_MODEL=dmtx/qwen3.5-9b-abliterated
GENERATION_MODEL_LITE=richardyoung/qwen3-4b-instruct-2507-abliterated
IDLE_TIMEOUT_MINUTES=10

Ingest notes

Put your markdown files in data/ (any subdirectory structure works), then:

cd src
python ingest.py

Resumable — already-indexed chunks are skipped on re-run.

Query

CLI:

python query.py "your question here"
python query.py --lite "your question here"   # use 4B model
python query.py --prose "your question here"  # narrative answer format

Web UI:

python api.py
# open http://localhost:8000

Docker deployment

Runs Qdrant + API server together. Ollama must be running on the host.

docker compose up --build

The API is available at http://localhost:8000. Data directory is mounted read-only; history.db is mounted for persistence.

Telegram scraper (optional)

To ingest a Telegram channel into data/telegram/:

1. Get your API credentials at my.telegram.org
2. Set TG_API_ID, TG_API_HASH, TG_CHANNEL in .env
3. Run:

cd src
python scrape_telegram.py

To find a channel's numeric ID:

python list_dialogs.py

Then re-run ingest.py to index the new files.

Project structure

src/
  api.py            # FastAPI app, SSE streaming
  query.py          # full retrieval pipeline, CLI
  ingest.py         # chunk, embed, upsert to Qdrant
  config.py         # all settings
  db.py             # SQLite history
  scrape_telegram.py
  list_dialogs.py
  static/index.html # web UI
data/               # your markdown files (gitignored)

Configuration reference

All settings are in src/config.py and can be overridden via environment variables:

Variable	Default	Description
QDRANT_URL	http://localhost:6333	Qdrant address
EMBED_MODEL	bge-m3	Ollama embedding model
GENERATION_MODEL	dmtx/qwen3.5-9b-abliterated	Default LLM
GENERATION_MODEL_LITE	richardyoung/qwen3-4b-instruct-2507-abliterated	Lite LLM
RERANKER_MODEL	BAAI/bge-reranker-v2-m3	HuggingFace reranker
IDLE_TIMEOUT_MINUTES	10	API idle shutdown