Feature: Knowledgebase RAG System — User-Configured Document Directory with Local Embedding, Hybrid Search & Auto-Retrieval

[teknium1]

Overview

Hermes Agent needs a knowledgebase RAG system that lets users point to a directory (or multiple directories) of documents and have the agent automatically index, embed, and retrieve relevant content during conversations. This is the retrieval pipeline that powers the workspace concept described in #531 and complements the semantic code search in #489.

The core UX: set a directory in config, the agent indexes it, and relevant knowledge is automatically surfaced when you ask questions — no manual @ references or file paths needed.

# ~/.hermes/config.yaml
knowledgebase:
  enabled: true
  directories:
    - ~/notes
    - ~/work/docs
    - ~/.hermes/workspace
  auto_retrieve: true          # inject relevant chunks into context automatically
  max_context_chunks: 8        # max chunks to inject per query
  embedding_model: local       # "local" (default) or "openai" or "ollama"
  reindex_on_change: true      # watch for file changes

---

Competitive Landscape — How Every Major Agent Does This

Category A: Full RAG with Vector Search

Agent	Embedding	Vector Store	Chunking	Retrieval Trigger
Cursor	Proprietary code model	Turbopuffer (cloud)	Tree-sitter AST for code, ~500 tokens for docs	Auto + @Codebase/@Docs
Windsurf	Proprietary "M-Query"	Cloud (single-tenant)	Proprietary	Automatic via context engine
Continue.dev	all-MiniLM-L6-v2 (local) or voyage-code-3	LanceDB (embedded, ~/.continue)	AST-based + fixed-length	Agent tools (deprecated @codebase)
Roo Code	Configurable (Gemini free, OpenAI, Ollama)	Qdrant (Docker or cloud)	Tree-sitter, 100-1000 chars	Auto via codebase_search tool
Augment Code	Custom "helpfulness" model	Custom (BigTable/GCP)	Proprietary, seconds-latency updates	Fully automatic

Category B: Agentic Exploration (No RAG)

Agent	Approach	Tradeoff
Claude Code	grep/find/ripgrep + CLAUDE.md context files	~40% more token consumption than RAG
Cline	Traces imports, reads files directly	Simpler but slower on large codebases
Aider	AST repo map with PageRank-like ranking	Clever middle ground, no embeddings
OpenHands	Agentic shell exploration + AGENTS.md	RAG on their roadmap as high priority
Bolt.new	Entire project in context window	Only works for small projects

Category C: Document/Knowledge Indexing

Framework	Approach	Stars
LlamaIndex	SimpleDirectoryReader → VectorStoreIndex → QueryEngine	40.8k
LangChain	DirectoryLoader → TextSplitter → RetrievalQA	105k
RAGFlow	Deep document understanding (tables, layouts, PDFs)	48.5k
Mem0	Intelligent memory layer with contradiction resolution	27.3k
txtai	All-in-one embeddings DB + workflow engine	10.7k

Key Takeaway

The field is split ~50/50 between RAG-based and agentic-exploration approaches. Hermes currently sits firmly in Category B (agentic exploration via search_files + read_file). Adding RAG gives users the best of both worlds — fast semantic retrieval for knowledge bases plus precise agentic exploration for code. MCP is emerging as the standard bridge protocol, but a native implementation is more reliable and lower-latency.

---

Recommended Technical Stack

Based on research into what works best for a single-user CLI agent that needs to be lightweight, fast, and dependency-minimal:

Embedding Model

Model	Params	Dims	Speed	Quality	Notes
all-MiniLM-L6-v2	22M	384	14.7ms/1K tok	Good	Best speed, CPU-friendly
BGE-base-en-v1.5	110M	768	22.5ms/1K tok	Better	Best quality/size ratio
CodeRankEmbed	137M	—	—	Best for code	MIT, 8192 context
nomic-embed-text	137M	768	Via Ollama	Good	Fully open, MoE variant available
EmbeddingGemma-300M	300M	128-768	<22ms	Good	Matryoshka dims, 2048 context

Recommendation: Default to all-MiniLM-L6-v2 via fastembed (pure Python, no Ollama/server needed). Offer ollama and openai as alternatives for users who want better quality or already have those set up.

Vector Store

Store	Type	Hybrid Search	Storage	Performance
sqlite-vec	SQLite extension	YES (with FTS5)	Single .sqlite file	Brute-force, good to ~100K vectors
LanceDB	Embedded Rust	Manual	Lance files	Sub-100ms at 1B vectors
ChromaDB	Embedded Python	No native BM25	SQLite + hnswlib	Easy API, more overhead
FAISS	In-memory library	No	Manual save/load	Fastest raw search

Recommendation: sqlite-vec + FTS5. Hermes already uses SQLite extensively (SessionDB). A single .sqlite file gives us:

• Vector similarity search via sqlite-vec extension
• BM25 keyword search via FTS5 (built-in to SQLite)
• Hybrid search by combining both scores
• Zero server processes, zero additional dependencies (beyond the extension)
• Natural fit with existing Hermes architecture

Fallback: ChromaDB for users who want HNSW performance at scale. LanceDB as a future option.

Chunking Strategy

For documents (Markdown, text, PDF):

• Recursive character splitting: 400-512 tokens, 10-20% overlap
• Separators hierarchy: ["\n\n", "\n", ". ", " ", ""]
• Preserve paragraph/section structure
• Store source file path + line range as metadata

For code files:

• Tree-sitter AST parsing (per Feature: Semantic Codebase Search — Tree-sitter + Embeddings as search_files target='semantic' Mode (inspired by Roo Code) #489)
• Split at function/class boundaries
• 400-600 token target per chunk
• Include docstrings with parent functions
• Contextual headers: # file: src/tools/search.py | class: SearchTool | method: search()

Key insight from research: Quality drops sharply above ~2500 tokens per chunk. Keep chunks concise (400-512) even with large context windows. AST-based chunking for code achieves 70.1% Recall@5 vs 42.4% for fixed-size — a massive difference.

Hybrid Search (Dense + Sparse)

Hybrid search improves recall 15-30% over either method alone. Critical for code (variable names, function names, error codes need exact BM25 matching) and docs (technical jargon + conceptual similarity).

Score = (alpha * cosine_similarity) + ((1 - alpha) * bm25_score_normalized)
alpha = 0.6  (slightly favor semantic for natural language queries)

Alternative: Reciprocal Rank Fusion (RRF) — more robust, no score normalization needed:

RRF_score = sum(1 / (k + rank_i)) for each retriever

Token Budget Management

For a 128K context agent:

• Retrieval budget: ~8,000-16,000 tokens (20-30% of window)
• At 512 tokens/chunk: 15-30 chunks max
• But quality peaks at 5-10 highly relevant chunks
• Apply relevance threshold (cosine > 0.5) — discard low-relevance even if under budget
• Deduplicate: remove chunks with >70% content overlap

---

Implementation Plan

Phase 1: Core RAG Pipeline (MVP)

Config:

knowledgebase:
  enabled: false               # opt-in
  directories: []              # user sets paths
  embedding_model: local       # "local" (fastembed), "openai", "ollama"

New files:

• tools/knowledgebase_tool.py — kb_search(query, directory?), kb_index(directory?), kb_status()
• agent/knowledgebase.py — Indexing engine: chunking, embedding, storage, retrieval
• agent/chunker.py — Document chunking (recursive text splitter + basic code splitter)

Index storage: ~/.hermes/knowledgebase/indexes/{dir-hash}.sqlite — one SQLite file per indexed directory containing:

• chunks table: id, file_path, chunk_index, content, start_line, end_line, file_hash
• embeddings virtual table (sqlite-vec): vector embeddings
• chunks_fts virtual table (FTS5): full-text search index
• files metadata table: path, mtime, size, content_hash, last_indexed

Supported file types (Phase 1):

• Markdown (.md), Text (.txt), reStructuredText (.rst)
• Python (.py), JavaScript (.js), TypeScript (.ts), JSON, YAML
• PDF (via pdftotext or pymupdf — optional dependency)

Indexing flow:

1. Walk configured directories (respect .gitignore patterns)
2. Compare file hashes against stored metadata — skip unchanged files
3. Chunk changed/new files (recursive text split for docs, basic AST for code)
4. Generate embeddings (batch, 60 chunks at a time)
5. Store in SQLite (sqlite-vec for vectors, FTS5 for text)
6. Update file metadata

Search flow:

1. Agent calls kb_search(query="deployment architecture")
2. Generate query embedding
3. Run hybrid search: sqlite-vec cosine similarity + FTS5 BM25
4. Score fusion (alpha=0.6 semantic, 0.4 keyword)
5. Return top-K chunks with file path, score, content, line range

CLI commands:

• /kb index [directory] — trigger indexing
• /kb search <query> — manual search
• /kb status — show indexed directories, file counts, last indexed time
• /kb clear [directory] — clear index for a directory

Phase 2: Auto-Retrieval & Smart Injection

Auto-retrieval: When enabled, before the agent sees the user's message, run a background kb_search and inject relevant chunks as a system context block:

[Knowledgebase Context — auto-retrieved, relevance-ranked]
From ~/notes/deployment.md (lines 12-45, score: 0.87):
  <chunk content>

From ~/work/docs/architecture.md (lines 1-30, score: 0.82):
  <chunk content>

Smart injection rules:

• Only inject if top chunk score > 0.5 (relevance threshold)
• Max 5-8 chunks, ~4000 tokens budget
• Deduplicate overlapping chunks
• Skip if query is clearly not knowledge-seeking (e.g., "fix this bug" with code context)

File watching:

• Use content-hash based change detection on session start
• Optional: watchdog library for real-time file watching (background process)
• Debounce changes (2 second window)

Additional config:

knowledgebase:
  auto_retrieve: true
  relevance_threshold: 0.5
  max_context_chunks: 8
  max_context_tokens: 4000
  watch_for_changes: false     # real-time reindexing

Phase 3: Advanced Features

• Tree-sitter code chunking (integrate with Feature: Semantic Codebase Search — Tree-sitter + Embeddings as search_files target='semantic' Mode (inspired by Roo Code) #489 work)
• Multi-model support: Switch between fastembed, Ollama (nomic-embed-text), OpenAI (text-embedding-3-small), Gemini
• Incremental reranking: Use a cross-encoder or cheaper LLM to rerank retrieved chunks
• Source citations: Include [Source: ~/docs/file.md:12-45] in agent responses
• Per-directory config: Different chunk sizes, embedding models per directory
• Conversation-aware retrieval: Use recent conversation context to refine search queries
• @kb reference syntax: Let users explicitly query knowledgebase in messages
• Gateway integration: /kb commands in Telegram/Discord/Slack
• MCP server: Expose knowledgebase as an MCP tool for other agents

---

Architecture Decisions

Why sqlite-vec over ChromaDB/LanceDB?

1. Single file — one .sqlite per indexed directory, trivially portable and backupable
2. Built-in hybrid search — FTS5 gives us BM25 for free alongside vector search
3. Hermes already uses SQLite — SessionDB, state.db, session storage all use SQLite
4. Zero server — no Docker, no background process, no port conflicts
5. Good enough performance — brute-force is fine for <100K vectors (most personal knowledge bases)
6. Familiar tooling — debugging is just sqlite3 index.sqlite

Why fastembed as default?

1. Pure Python — pip install fastembed, no Ollama server, no API key
2. CPU-friendly — all-MiniLM-L6-v2 runs in ~15ms per chunk on CPU
3. Tiny — 22M params, ~80MB download
4. No configuration — works out of the box
5. Upgradeable — users can switch to Ollama/OpenAI for better quality anytime

Why not just use MCP?

MCP could wrap a RAG server, but:

1. Adds a server process to manage (startup, crashes, restarts)
2. Latency overhead from IPC/HTTP
3. Configuration split between Hermes config and MCP server config
4. Native integration allows auto-retrieval (injecting context before the agent sees the message)
5. MCP can't participate in prompt building — it's a tool, not a context source

Native RAG with MCP exposure (Phase 3) gives the best of both worlds.

Context Injection vs Tool-Only

Both approaches have merit:

Approach	Pros	Cons
Auto-inject (Continue/Cursor style)	Zero friction, agent always has context	May inject irrelevant content, token waste
Tool-only (Roo Code style)	Agent decides when to search, precise	Agent may forget to search, extra tool call latency
Hybrid (recommended)	Auto-inject high-confidence matches, tool for explicit queries	More complex

Recommendation: Hybrid. Auto-inject when score > 0.7 (high confidence). Expose kb_search tool for explicit queries. User configurable.

---

Dependency Impact

Required (Phase 1)

• fastembed — embedding generation (~80MB model download on first use)
• sqlite-vec — SQLite vector extension (pip installable)

Optional

• pymupdf or pdftotext — PDF text extraction
• python-docx — DOCX text extraction
• watchdog — file system watching
• tree-sitter + language grammars — AST-based code chunking (Phase 3)
• ollama — local embedding via Ollama server
• openai — cloud embedding via OpenAI API

Installation

pip install hermes-agent[rag]           # fastembed + sqlite-vec
pip install hermes-agent[rag,docs]      # + PDF/DOCX support

---

Relationship to Other Issues

• Feature: User Workspace & Knowledge Base — Persistent Document Storage, Search, and RAG Integration #531 (User Workspace): This issue provides the RAG engine for Feature: User Workspace & Knowledge Base — Persistent Document Storage, Search, and RAG Integration #531's Phase 3. The workspace is the "where," this is the "how."
• Feature: Semantic Codebase Search — Tree-sitter + Embeddings as search_files target='semantic' Mode (inspired by Roo Code) #489 (Semantic Codebase Search): Shares chunking and embedding infrastructure. Code-focused search can reuse the same pipeline with tree-sitter chunking.
• Feature: Structured Memory System — Typed Nodes, Graph Edges, and Hybrid Search #346 (Structured Memory): Memory is agent-generated facts; knowledgebase is user-curated documents. Different sources, potentially shared retrieval.
• Feature: Configurable Embedding Infrastructure — Local (fastembed) + API (OpenAI) with Config Flag #675 (Embedding Infrastructure): Closed, but the embedding provider abstraction designed there feeds directly into this.

---

Open Questions

1. Auto-index on startup? Should the agent re-check file hashes on every session start, or only on explicit /kb index? Recommendation: check on startup if reindex_on_change: true, but make the hash check fast (<100ms for ~1000 files).

2. Index size limits? Should we cap the number of files/chunks per directory? Large directories (node_modules, .git) should be auto-excluded via .gitignore patterns.

3. Multi-user (gateway)? In messaging mode with multiple users, should each user have their own knowledgebase index, or share one? Recommendation: single shared index (matches current single-workspace design).

4. Embedding model lock-in? If a user changes embedding models, existing indexes become invalid. Should we auto-detect and re-index, or warn? Recommendation: store model name in index metadata, warn + offer re-index on mismatch.

5. Privacy for cloud embeddings? When using OpenAI/Gemini embeddings, document content is sent to their API. Should we warn users? Recommendation: yes, print a notice on first index with cloud provider.

---

References

• Cursor codebase indexing: tree-sitter AST → proprietary embeddings → Turbopuffer, Merkle tree sync
• Continue.dev custom RAG guide: voyage-code-3 + LanceDB + MCP server pattern
• Roo Code codebase indexing: tree-sitter → configurable embeddings → Qdrant, score threshold tuning
• Claude Code: deliberately no RAG, compensates with agentic exploration (~40% more tokens)
• Aider repo map: AST graph + PageRank ranking (no embeddings)
• sqlite-vec: https://github.com/asg017/sqlite-vec (SQLite vector extension)
• fastembed: https://github.com/qdrant/fastembed (lightweight embedding)
• BM25S: 500x faster than rank_bm25, pure numpy/scipy
• Research: AST chunking achieves 70.1% Recall@5 vs 42.4% for fixed-size (code-chunk benchmarks)
• Research: Hybrid search improves recall 15-30% over single-method retrieval
• Research: Quality peaks at 5-10 chunks; degrades above 2500 tokens/chunk

[m13v]

we built exactly this - a knowledgebase system that ingests user documents, embeds them locally, and retrieves with hybrid search. some things from production use:

for the embedding pipeline: batch ingestion matters more than model choice. we process browser history, documents, and captures in bulk, generating embeddings with all-MiniLM-L6-v2 (384-dim). SQLite + sqlite-vec stores everything locally. batch 50-100 items at a time - single-item embedding is 5-10x slower.

hybrid search is essential. we combine FTS5 keyword search (BM25 scoring) with sqlite-vec cosine similarity. weighted blend: 0.3 BM25 + 0.7 semantic. pure vector search misses exact terms (API names, error codes), pure keyword misses paraphrases.

for the auto-retrieval trigger: we use a lightweight index file (under 200 lines) that's always loaded. it contains brief descriptions of what's in the knowledge base. the agent reads the index, decides if deeper retrieval is needed, then queries specific memory files. this avoids stuffing the full knowledge base into every conversation.

the self-ranking piece was the biggest win: memories that aren't accessed decay in relevance over time. this prevents old, stale knowledge from dominating search results as the database grows. without compaction, retrieval quality degrades noticeably after a few thousand entries

[m13v]

our local knowledge base with hybrid search: https://github.com/m13v/user-memories - SQLite + FTS5 + sqlite-vec, browser data ingestion, self-ranking entries. the embedding logic: https://github.com/m13v/user-memories/blob/main/user_memories/embeddings.py. the DB schema: https://github.com/m13v/user-memories/blob/main/user_memories/db.py

[m13v]

We built something similar for our user-memories npm package - a local knowledge extraction system that pulls from browser history, bookmarks, and documents, then indexes everything with embeddings into SQLite with semantic search.

A few things we learned that might help here:

1. For the embedding model choice, we went with a similar approach to your all-MiniLM-L6-v2 recommendation. The key insight was that for personal/local knowledge bases, embedding quality matters less than you'd think - the corpus is small enough that even mediocre embeddings give good retrieval. Speed and zero-server-dependency wins.

2. sqlite-vec + FTS5 is the right call. We use SQLite as the single store for both vector search and full-text search. Hybrid scoring (combining cosine similarity with BM25) was the biggest quality improvement we made. Pure vector search misses exact keyword matches, pure BM25 misses semantic similarity.

3. One thing missing from the proposal - deduplication across directories. When users point multiple directories at the system, you get overlapping content (same README referenced from multiple places, symlinks, etc.). We hash content chunks before embedding to avoid duplicate vectors.

4. The auto_retrieve: true with max_context_chunks is important to get right. We found that injecting too many chunks (even relevant ones) actually degrades response quality. 5-8 chunks is the sweet spot. Beyond that, the model starts getting confused by contradictory or redundant information.

5. For reindex_on_change - fsnotify/watchman works but be careful with debouncing. Editors like VS Code trigger dozens of file events per save. We batch file change events with a 2-second debounce window before re-indexing.

[m13v]

For reference, here's our implementation of the hybrid search with SQLite embeddings: https://github.com/m13v/user-memories/blob/main/user_memories/embeddings.py

And the database layer that handles dedup + FTS5 indexing: https://github.com/m13v/user-memories/blob/main/user_memories/db.py

The browser history ingestion pipeline (which deals with similar directory-watching and incremental indexing problems): https://github.com/m13v/user-memories/blob/main/user_memories/ingestors/history.py

[kshitijk4poor]

Expanded the parsing/tool-only PR so it now covers more of the LiteParse capability surface while still staying out of retrieval/indexing scope.

Additional changes now on PR #2082:

• document_parse supports OCR-related overrides (ocr_enabled, ocr_server_url, ocr_language)
• page-limited parsing via target_pages
• DPI and precise-bbox controls
• optional text item and bounding box output
• PDF page screenshot generation for downstream vision workflows
• optional Hermes skill: optional-skills/research/document-parse/SKILL.md

That keeps this PR focused on document parsing as a first-class local primitive while making it much more useful as a foundation for future workspace ingestion and knowledgebase work.

[m13v]

nice progress on the parsing side. the OCR overrides and page-limited parsing are practical additions - for large PDFs, being able to target specific pages avoids a lot of wasted compute. the screenshot generation for downstream vision workflows is a good call too, we've found that combining extracted text with page images gives much better results than either alone for complex layouts.