engramdb

An embedded vector database in pure Rust, with built-in document chunking and hybrid (dense + BM25) retrieval — plus engram, a terminal app for retrieval-augmented chat over your documents with a local Ollama model.

No server, no external services. Just a crate (and an optional TUI).

document ──▶ chunker ──▶ embedder ──▶ index ──▶ hybrid search
(any type)   (smart       (pluggable)  (flat or   (dense + BM25,
              splitting)               HNSW)       RRF fusion, MMR)

Features

• Chunking for every document type — plain text, Markdown (heading-aware), HTML, CSV/TSV (record-aware), JSON (structure-flattening), source code (definition-aware), and PDF (with OCR fallback for scanned pages). A shared recursive splitter gives consistent, overlap-aware chunk sizing across formats.
• Two indexes behind one trait
  • FlatIndex — exact brute-force, rayon-parallel, perfect recall.
  • HnswIndex — approximate nearest neighbour (HNSW) with Malkov's diversity heuristic for high recall and sub-linear queries.
• Hybrid retrieval — dense (vector) and sparse (BM25) rankings fused with reciprocal-rank fusion, per-arm relevance floors, optional MMR reranking for result diversity, and a pluggable cross-encoder rerank hook.
• Deletes & updates — stable u64 ids decoupled from storage slots; remove, remove_source, update_text, and compact to reclaim tombstones.
• int8 quantization — optional scalar quantization for ~4× less memory at negligible recall cost.
• Thread-safe — VectorDb is Sync; share via Arc. Searches run concurrently (read lock); embedding happens outside the lock.
• Pluggable embeddings — implement the Embedder trait over any model. Built in: HashingEmbedder (deterministic, dependency-free) and SentenceTransformer (real transformer embeddings via candle, behind the candle feature). The TUI adds an Ollama embedder.
• Efficient core — contiguous row-major vector storage, auto-vectorised f32 and fused int8 distance kernels, epoch-stamped HNSW visited sets, bounded parallel top-k, batched inference.
• Persistence — save/load the whole database to a single file, written atomically (temp file + rename).

Quick start (library)

[dependencies]
engramdb = { git = "https://github.com/sxtj/engramdb", features = ["full"] }

use engramdb::{VectorDb, IndexKind, Metric, HashingEmbedder};

let embedder = HashingEmbedder::new(384);
let db = VectorDb::new(embedder, Metric::Cosine, IndexKind::Hnsw);

db.add_file("README.md")?;                          // type inferred from extension
db.add_text("Rust is fast and safe.", "note")?;     // raw text

for hit in db.search("memory safety", 5)? {
    println!("{:.3}  {}  {}", hit.score, hit.chunk.source, hit.chunk.text);
}

db.save("index.vdb")?;
let db = VectorDb::load(HashingEmbedder::new(384), "index.vdb")?;

Run the demo: cargo run --example ingest

HashingEmbedder captures lexical overlap, not meaning. For semantic search use the transformer embedder (--features candle):

use engramdb::{VectorDb, IndexKind, Metric, SentenceTransformer};

let embedder = SentenceTransformer::new()?;   // all-MiniLM-L6-v2, cached on first use
let db = VectorDb::new(embedder, Metric::Cosine, IndexKind::Hnsw);
db.add_text("A dog is chasing a ball across the park.", "doc1")?;
let hits = db.search("a puppy plays fetch outdoors", 3)?;   // zero shared keywords

Or bring your own model:

use engramdb::{Embedder, Result};

struct MyModel;
impl Embedder for MyModel {
    fn dim(&self) -> usize { 768 }
    fn embed(&self, text: &str) -> Result<Vec<f32>> { todo!("run inference") }
    // override embed_batch for true batched inference
}

Terminal UI (engram)

Retrieval-augmented chat over your documents with a local Ollama model. It starts ollama serve for you if it isn't already running.

ollama pull nomic-embed-text      # embedding model (locked per collection)
ollama pull qwen2.5:7b            # any chat model you like

cargo install --path . --features tui   # installs `engram` to ~/.cargo/bin
ENGRAM_CHAT_MODEL=qwen2.5:7b engram

Tabs (Tab / Shift+Tab): Chat (streamed answers with [n] citations and a Sources panel), Library (i ingest file/folder, d delete), Collections (separate corpora so topics can't collide), Models (switch chat model live), Stats, Logs. Ctrl+Q quits and saves.

Env config: OLLAMA_HOST, ENGRAM_CHAT_MODEL, ENGRAM_EMBED_MODEL, ENGRAM_COLLECTION.

Retrieval design

Retrieval fuses dense and sparse arms with RRF, then MMR-reranks. Each arm has a relevance floor so an off-topic question retrieves nothing rather than tangential chunks:

• ENGRAM_DENSE_FLOOR (default 0.4) — minimum cosine similarity.
• ENGRAM_SPARSE_FLOOR (default 1.0) — minimum BM25 score. IDF-weighted, so rare-term matches (identifiers, error codes) clear it; stray common-word overlap does not.
• ENGRAM_HYBRID_GATE (default on) — BM25 may only re-rank chunks dense already found relevant, never inject off-topic ones. Set 0 for full lexical recall.
• ENGRAM_SPARSE_WEIGHT (default 1.0) — down-weight the lexical arm.
• ENGRAM_LLM_RERANK=1 — add an LLM cross-encoder rerank pass.

When retrieval comes back empty the model answers from general knowledge but says so; [n] citations are validated against the retrieved sources, so a hallucinated reference renders as plain text, not a fake citation.

Diagnose retrieval headless — when a query returns "that isn't in your documents" and you want to know why:

engram query "how does the ? operator work" -k 5

prints every candidate per arm with its raw score and whether it cleared the floor, then the fused result (--json for a machine-readable trace, --answer to also stream the reply). Compare strategies offline with cargo run --example eval_retrieval (recall@k / MRR: dense vs hybrid vs MMR).

PDF support (pdfium) and OCR

PDF extraction prefers pdfium (Google's PDF engine) and falls back to the pure-Rust pdf-extract. Fetch the prebuilt library once:

./scripts/fetch-pdfium.sh           # downloads libpdfium into ./lib

engram looks for pdfium in $ENGRAM_PDFIUM_PATH, beside the binary, then ./lib, then the system library. Scanned/image-only PDFs are OCR'd automatically when the tesseract CLI is on PATH (brew install tesseract); OCR'd chunks are flagged in the Sources panel. Disable with ENGRAM_OCR=0; set language via ENGRAM_OCR_LANG (default eng).

Build flags

Always use --release (debug builds run the search math unoptimized). For the fastest indexing/search let the kernels use your CPU's SIMD:

RUSTFLAGS="-C target-cpu=native" cargo run --release --features tui --bin engram

Cargo features

Feature	Adds	Default
markdown	Markdown parsing	✅
html	HTML → text	✅
csv	CSV/TSV parsing	✅
pdf	PDF text extraction (pdfium + pdf-extract)	❌
ocr	OCR fallback for scanned PDFs (pdfium render + tesseract)	❌
candle	Transformer embeddings	❌
full	markdown + html + csv + pdf	❌
tui	engram terminal app (Ollama RAG chat); includes ocr	❌

Disabled parsers fall back to plain-text splitting (except PDF, which errors).

Persistence & storage

The whole database — index, vectors, chunk payloads — serializes to one bincode file, written atomically; the TUI adds a JSON manifest namespaced by embedding model, so an index built with one embedder can never be loaded against another. Everything loads into RAM: simple, fast, and fine up to millions of chunks. Beyond that, the next step is an mmap'd store with append-only segments (see IMPROVEMENTS.md).

Tuning

• Chunk size / overlap — ChunkConfig / SplitConfig (SplitConfig::from_tokens(512, 64) to size in approximate tokens).
• HNSW — HnswConfig { m, ef_construction, ef_search, .. }. Raise ef_search for recall, lower for speed.
• Metric — Metric::Cosine (auto-normalised), Dot, or L2.
• Memory — Quantization::Int8 for ~4× smaller vectors.

License

MIT — see LICENSE.

---

Built with AI-assisted development using Claude Code. All code was reviewed, tested, and modified as needed.