docsearch

A self-hosted document search server — ingest a personal library of documents and audio, make it searchable with a local retrieval pipeline, and plug it into ChatGPT as a custom connector so it can answer questions from the corpus with citations. Everything — parsing, OCR, transcription, embeddings, reranking, the default chat model — runs on your own hardware. The only thing that ever leaves the box is the handful of passages a search returns.

It's built to be usable by a non-technical person: the console speaks in folders, documents, and passages rather than buckets, chunks, and embeddings, failures explain themselves, and every document page has a "will a search actually find this?" retrieval test.

How it works

• Ingestion (docsearch/ingest.py, transcribe.py, units.py, split.py) — Docling parses documents (docx/pdf/pptx/txt/md/html) and auto-routes scanned pages through OCR; faster-whisper transcribes audio. Anthology-style PDFs are detected and physically sliced into per-unit child sources so retrieval lands on the actual unit, not a 400-page blob. Text is chunked with the embedder's own tokenizer (HybridChunker) and embedded with a served Qwen3-Embedding-0.6B.
• Store — Postgres + pgvector (HNSW), plus a full-text index and unaccent-folded title search. Original files and GPU-expensive transcript sidecars live on disk under data/storage/.
• Retrieval (docsearch/query.py) — one funnel: vector nearest-neighbor (plus an optional lexical leg), a served Qwen3-Reranker-0.6B cross-encoder, and reciprocal-rank fusion. Console, MCP server, CLI, and the eval harness all call the same query.search.
• Console (app/) — a password-gated FastAPI + HTMX web app: drag-in uploads with background processing, a folder-style library browser, Ask (one-shot question → grounded, cited answer with the raw retrieval on display) and Talk (durable multi-turn conversations where the model drives the real MCP search/fetch tools itself).
• MCP server (docsearch/server.py) — a FastMCP search/fetch server in OpenAI's connector shape, exposed through an HTTPS tunnel at a secret path. Every result carries a citation URL that resolves to a public /doc/{id} reading page.

Prerequisites

• Python 3.12+ and uv.
• Docker — the Postgres + pgvector database runs in a container (db.compose.yml: image pgvector/pgvector:pg17, published on host port 5438).
• An OpenAI-compatible inference endpoint for embeddings, reranking, and chat (the author runs vLLM behind a local litellm proxy at http://127.0.0.1:4000/v1; any endpoint speaking the same API works). docsearch uses three routes:
  • embeddings → Qwen/Qwen3-Embedding-0.6B (1024-dim) — required for ingest & search.
  • rerank → Qwen/Qwen3-Reranker-0.6B — the cross-encoder (can be disabled).
  • qwen-chat → a local chat model — the console's default answer generator.
• ffmpeg on PATH (audio normalization for transcription).
• A CUDA GPU is used by Docling OCR and the served models; faster-whisper runs on CPU by default to keep VRAM free.

Setup

uv sync                       # create the venv, install docsearch (editable) + deps
cp .env.example .env          # then fill in LLM_API_KEY and any secrets
docker compose -f db.compose.yml up -d
uv run python -m docsearch.db # bootstrap the schema (idempotent) + connectivity check

docsearch/config.py loads .env once; DATABASE_URL and LLM_BASE_URL/LLM_API_KEY are required. The schema is additive and idempotent, so python -m docsearch.db is safe to re-run.

Running

Service	Command	Default address
Console (web app)	uv run uvicorn app.api:app --host 127.0.0.1 --port 8080	http://127.0.0.1:8080
MCP server (for ChatGPT)	uv run python -m docsearch.server	http://0.0.0.0:8000/mcp

Set DOCSEARCH_PASSWORD in .env to gate the console; unset, it runs open and warns at startup. The MCP server exposes GET /healthz for monitoring; the console /system page shows live health of the DB, embedder, reranker, and MCP server. For always-on serving, run both under systemd (or your process manager of choice) — infra.toml holds the author's service definitions as a reference.

Connecting ChatGPT

The MCP server implements OpenAI's connector search/fetch schema. To wire it up:

1. Expose the MCP server over public HTTPS (ngrok / Tailscale Funnel / Caddy). Set PUBLIC_BASE_URL in .env to that base URL (it is also used for the /doc/{id} citation reading pages).
2. Set MCP_PATH_SECRET to a long random string. ChatGPT connectors can't send a bearer token, so the endpoint moves from /mcp to /<secret>/mcp as minimal auth. Restart the server; the endpoint is then PUBLIC_BASE_URL/<secret>/mcp.
3. In ChatGPT (Plus, Developer Mode): Settings → Connectors → add a custom connector pointing at that URL. The console's /system page prints the exact URL to paste (with a copy button; the secret is masked on screen).

The console's Ask page has a connector-simulation mode that drives the real MCP server the same way ChatGPT will, so you can rehearse the tool loop locally before pointing ChatGPT at it.

Daily use

Normal use is entirely through the console:

• Add (/upload) — drag in files (single or batch), pick a folder and origin, and optionally process immediately. Processing runs in the background; failures surface with a reason.
• Library (/sources) — browse folder-style, search by title, open a document to edit metadata, inspect its passages, run a "will a search find this?" retrieval test, download the original, or delete it.
• Ask (/chat) — one-shot: real retrieval, a grounded cited answer, and the retrieved passages shown for inspection.
• Talk (/talk) — multi-turn: conversations persist, follow-ups like "what about the second one?" resolve against the history, and older turns fold into a visible rolling summary as they scroll out of the model's window.

CLI cookbook

Task	Command
Bootstrap / migrate schema	uv run python -m docsearch.db
Query the corpus (tuning)	uv run python -m docsearch.query "mindfulness of breathing" --k 5
Query with a filter	uv run python -m docsearch.query "..." --bucket talks --provenance own
Upload files (CLI)	uv run python -m docsearch.upload <files|folder> --bucket books --credit "…" --reference --ingest
Ingest pending sources	uv run python -m docsearch.ingest --pending
Re-ingest one source	uv run python -m docsearch.ingest --source <id>
Force OCR (broken PDF text layer)	uv run python -m docsearch.ingest --source <id> --force-ocr
Delete a source	uv run python -m docsearch.manage --delete <id>
Delete a collection	uv run python -m docsearch.manage --delete-collection <bucket> --collection "…"
Rebuild the vector index after bulk deletes	uv run python -m docsearch.manage --reindex
Score retrieval quality	uv run python -m docsearch.eval
Gate against the regression baseline	uv run python -m docsearch.eval --compare reports/baseline.json

Eval + regression workflow

reports/eval.json is the ground-truth query set (real-corpus queries → expected source_ids, with optional per-query bucket/provenance filters). docsearch.eval scores recall@k and MRR against the live DB. When you change anything that affects ranking (reranker, chunking, HYBRID_LEXICAL, candidate depth, RRF_K …), re-run and gate against the committed baseline:

uv run python -m docsearch.eval --compare reports/baseline.json   # exits non-zero on regression

reports/baseline.json is the tracked reference; reports/eval_results*.json are generated outputs (gitignored). To promote a new baseline after an intended improvement, dump a fresh report (--json) and replace baseline.json deliberately. (reports/expected.json is a different file — the toy seed-fixture acceptance set used by tests/test_pipeline.py, not this harness.)

Backup & restore

The corpus lives in exactly two single-copy places: the Postgres volume and data/storage/ (originals + GPU-expensive .transcript.json sidecars). Back both up:

./scripts/backup.sh            # → backups/<timestamp>/ragdb.dump + storage/

Cron it (see the header of scripts/backup.sh for a crontab line and the full restore procedure). The manual equivalents:

# dump
pg_dump "$DATABASE_URL" -Fc -f ragdb.dump
rsync -a data/storage/ /somewhere/safe/storage/
# restore (into an empty ragdb) then rebuild the index
pg_restore --clean --if-exists --no-owner -d "$DATABASE_URL" ragdb.dump
rsync -a --delete /somewhere/safe/storage/ data/storage/
uv run python -m docsearch.manage --reindex

Project layout

Path	What
docsearch/	The runtime library (installed package).
docsearch/config.py	Single source of all environment configuration.
docsearch/db.py	Connection helper + idempotent schema bootstrap.
docsearch/embed.py, rerank.py	Served embedder + cross-encoder clients.
docsearch/query.py	The retrieval funnel (vector + optional FTS → rerank → RRF).
docsearch/ingest.py, transcribe.py, units.py, split.py	Docling ingest, ASR, anthology detection + slicing.
docsearch/upload.py, manage.py	Intake and delete/edit/reindex operations.
docsearch/eval.py	Retrieval eval + regression gate.
docsearch/server.py	FastMCP search/fetch server + public /doc pages.
app/	The FastAPI + Jinja2 + HTMX console (imports docsearch.*).
app/conversation.py, app/context.py	Multi-turn Talk: persistence, tool loop, window management.
scripts/	One-off tools + corpus builders + backup.sh (see scripts/README.md).
tests/	pytest suite (unit + integration-marked).
docs/	Design narrative + the system map source (docs/archive/ = superseded).
reports/	Eval ground truth (eval.json, expected.json, baseline.json).
db.compose.yml, docker-compose.prod.yml	The Postgres container; the slim prod MCP image.

Testing

uv run pytest -q                          # full suite (needs DB + inference endpoint)
uv run pytest -q -m "not integration"     # fast unit tests only (fusion math, helpers, escaping)
uv run ruff check . && uv run ruff format --check .

Integration tests run against a throwaway ragdb_test database and a temp storage dir (tests/conftest.py) — they never touch the live corpus.

Troubleshooting

• Ingest or search fails with a connection/timeout error — the inference proxy or a served model is down. Check your embedding/chat endpoint is up and LLM_BASE_URL points at it; the console /system page shows each dependency's health. Search falls back to vector-only order with a warning if only the reranker is unreachable (set RERANK_ENABLED=0 to disable it entirely).
• A diacritic-heavy title won't match in Library search — the unaccent extension powers ASCII-folded matching; make sure python -m docsearch.db has run (it creates it).
• Recall looks capped / a known-good chunk is missing on a filtered search — HNSW scans yield at most hnsw.ef_search rows; query.search raises it with headroom, and filtered queries flip on pgvector's iterative scan. After a large delete, run uv run python -m docsearch.manage --reindex (tombstoned graph nodes degrade recall).
• A born-digital PDF ingests as garbage (bad font/ToUnicode encoding) — re-ingest with --force-ocr; set meta.force_ocr on the source so re-ingests keep OCR-ing it.
• The local chat model returns an empty answer — keep DOCSEARCH_LOCAL_DISABLE_THINKING=1; a "thinking" model otherwise spends its whole context on hidden reasoning.

License

MIT