contextsearch 3.0.0

contextsearch 3.0.0 rewrites the version 2 proof of concept into a self-hosted search service that can be run as a server, used as a library and extended.

Highlights

• Hybrid search with evidence. BM25 over SQLite FTS5 plus embedding search, fused with reciprocal rank fusion, with an optional cross-encoder rerank. Every result says which retrievers found it and at what rank, which query terms matched, the cosine per model, a confidence label, and the page and character span in the source.
• Folders that stay in sync. Point it at a folder of PDF, DOCX, PPTX, text, Markdown, HTML or CSV files. New files are indexed within seconds, edited files get a new revision, deleted files leave the index. OCR for scanned pages and images is an optional extra.
• Several embedding models side by side, a Compare page, and contextsearch eval to score strategies on your own labelled queries (recall, MRR, nDCG, latency).
• Built to be operated. One process and one SQLite file, a REST API under /api/v1 with OpenAPI docs, API key auth, a folder allow-list, JSON logs with request ids, Prometheus metrics, a Dockerfile and a compose file.
• A new web UI with a page viewer that highlights the exact passage, light and dark themes, no build step and no external assets.

Upgrading from 2.x

The data written by version 2 cannot be read by version 3. Re-add the folder or re-upload the files. The default embedding model is now BAAI/bge-small-en-v1.5, scores are cosine similarities and BM25 values, the server listens on port 8080 and all endpoints moved under /api/v1. docs/DEPLOYMENT.md has the migration notes.

Install

git clone https://github.com/inboxpraveen/Context-Search-Engine.git
cd Context-Search-Engine
pip install -e ".[dense]"
contextsearch serve

The full list of changes is in CHANGELOG.md.