Skip to content

jkitchin/emacs-rag-libsql

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Emacs RAG with LibSQL

Overview

emacs-rag-libsql is a complete Retrieval-Augmented Generation (RAG) system designed for Emacs integration. It provides semantic search capabilities over your local documents using vector embeddings and advanced two-stage reranking for improved relevance.

The system consists of two main components:

  1. Python FastAPI Server (emacs-rag-server) - A REST API service providing document indexing, vector search, and reranking
  2. Emacs Lisp Package (emacs-rag) - An Emacs package for server management, file indexing, and search interface

Why This Matters

Traditional text search finds what you type. Semantic search finds what you mean.

  • Search for “machine learning algorithms” and find documents about “neural networks” and “deep learning”
  • Find relevant content even when different terminology is used
  • Navigate directly to the most relevant sections in your notes
  • Two-stage retrieval ensures both speed and accuracy

Features

🔍 Multiple Search Modes

  • Vector Search: Semantic similarity using embeddings for conceptual matching
  • Full-Text Search: Fast FTS5-powered keyword search with BM25 ranking
  • Hybrid Search: Combines vector and full-text search with configurable weighting
  • Org Heading Navigation: Jump directly to any org heading across all indexed files
  • Semantic Org Heading Search: Dynamic real-time semantic search across headings (with Ivy)
  • Configurable Models: Choose from multiple embedding models based on your needs

🎯 Two-Stage Reranking

Stage 1: Fast Bi-Encoder Retrieval
  ├─ Encode query → embedding vector
  ├─ Vector search → Top-K candidates (e.g., K=20)
  └─ Fast but approximate ranking

Stage 2: Precise Cross-Encoder Reranking
  ├─ Score each query-document pair directly
  ├─ Re-sort by cross-encoder scores
  └─ Return Top-N results (N=user limit)

This approach combines the speed of vector search with the accuracy of cross-encoder scoring.

📝 Smart Document Processing

  • Automatic Chunking: Documents split into overlapping chunks with configurable size
  • Line Number Tracking: Navigate directly to the exact line in your files
  • Metadata Support: Attach custom metadata (author, tags, etc.) to indexed documents
  • Batch Processing: Efficient embedding generation in batches
  • Multiple File Types: Extensible to support any text-based format (default: org-mode)

🔄 Seamless Emacs Integration

  • Auto-indexing: Automatically reindex files when you save them
  • Direct Navigation: Jump straight to relevant lines in your documents
  • Transient Menu: Beautiful, organized interface for all operations
  • Ivy Integration: Enhanced search result selection with dynamic collections (fallback to completing-read)
  • Real-time Search: Dynamic Ivy collections update results as you type
  • Async Operations: Non-blocking directory indexing
  • Server Lifecycle: Automatic server management - starts when needed
  • gptel Integration: LLM function calling tools for RAG-augmented AI interactions

🗄️ LibSQL Backend

  • SQL + Vectors: Combines the power of SQL with vector similarity search
  • Efficient Storage: Separate tables for documents and embeddings
  • Foreign Key Constraints: Data integrity with cascading deletes
  • Fallback Support: Works even without vector extension (slower but functional)
  • Local First: Your data stays on your machine

⚙️ Highly Configurable

All aspects are configurable through environment variables or Emacs customization:

  • Chunk size and overlap
  • Embedding models (sentence-transformers)
  • Reranking models (cross-encoder)
  • Search parameters
  • File extensions to index
  • Database location
  • Server settings

Architecture

System Overview

┌─────────────────────────────────────────────────────┐
│                    Emacs Client                      │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────┐ │
│  │   Server     │  │   Indexing   │  │   Search   │ │
│  │  Management  │  │   Commands   │  │  Interface │ │
│  └──────────────┘  └──────────────┘  └────────────┘ │
└─────────────────────────────────────────────────────┘
                          │
                    HTTP/REST API
                          │
┌─────────────────────────────────────────────────────┐
│              Python FastAPI Server                   │
│  ┌──────────────────────────────────────────────┐  │
│  │              API Routes                       │  │
│  │  /index  /search/vector  /search/text        │  │
│  │  /search/hybrid  /org-headings  /files       │  │
│  └──────────────────────────────────────────────┘  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────┐  │
│  │ File Service │  │Search Service│  │   Stats  │  │
│  └──────────────┘  └──────────────┘  └──────────┘  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────┐  │
│  │   Chunking   │  │  Embeddings  │  │ Reranker │  │
│  └──────────────┘  └──────────────┘  └──────────┘  │
│  ┌──────────────────────────────────────────────┐  │
│  │      LibSQL Database with Vector Storage     │  │
│  └──────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────┘

Database Schema

Documents Table

Stores text chunks with metadata and line tracking:

CREATE TABLE documents (
    id TEXT PRIMARY KEY,              -- {path}:{chunk_index}
    source_path TEXT NOT NULL,        -- Absolute file path
    chunk_index INTEGER NOT NULL,     -- 0-based chunk position
    line_number INTEGER NOT NULL,     -- Starting line (1-based)
    content TEXT NOT NULL,            -- Chunk text
    chunk_size INTEGER NOT NULL,      -- Actual character count
    chunk_total INTEGER NOT NULL,     -- Total chunks for this file
    metadata JSON,                    -- Custom metadata as JSON
    created_at INTEGER,
    updated_at INTEGER
);

Embeddings Table

Stores vector embeddings linked to documents:

CREATE TABLE embeddings (
    id TEXT PRIMARY KEY,              -- Same as documents.id
    vector BLOB NOT NULL,             -- Float32 vector
    model TEXT NOT NULL,              -- Embedding model identifier
    created_at INTEGER,
    FOREIGN KEY (id) REFERENCES documents(id) ON DELETE CASCADE
);

CREATE INDEX idx_embeddings_vector ON embeddings(vector) USING vector_cosine;

ML Models

Default Embedding Model

Model: sentence-transformers/all-MiniLM-L6-v2

  • Dimensions: 384
  • Size: ~80MB
  • Speed: Very fast inference
  • Quality: Good general-purpose semantic similarity
  • Training: MS MARCO passage ranking dataset

Default Reranker Model

Model: cross-encoder/ms-marco-MiniLM-L-6-v2

  • Size: ~90MB
  • Speed: Moderate (only applied to top-K candidates)
  • Quality: Significantly better than distance metrics alone
  • Training: MS MARCO passage reranking dataset

Installation

Prerequisites

  • Python 3.10 or higher
  • Emacs 27.1 or higher
  • uv (recommended) or pip for Python dependencies
  • transient package for Emacs (usually included with modern Emacs)

Install Python Server

# Navigate to server directory
cd emacs-rag-libsql/emacs-rag-server

# Install with uv (recommended)
uv sync

# Or install with pip
pip install -e .

# Verify installation
emacs-rag-server --help

Install Emacs Package

Add to your Emacs configuration:

;; Add to load path
(add-to-list 'load-path "/Users/jkitchin/Dropbox/emacs/user/emacs-rag-libsql/emacs-rag/")

;; Load the package
(require 'emacs-rag)

;; Optional: Set custom database path
(setq emacs-rag-db-path "/Users/jkitchin/Dropbox/emacs/cache/rag-database")

;; Optional: Configure indexed file extensions
(setq emacs-rag-indexed-extensions '("org" "txt" "md"))

;; Optional: Disable auto-indexing on save
(setq emacs-rag-auto-index-on-save nil)

How do I force it to reload after changing the files?

;; Load the specific file with full path
(load-file (expand-file-name "emacs-rag/emacs-rag-server.el" default-directory))
(load-file (expand-file-name "emacs-rag/emacs-rag-index.el" default-directory))
(load-file (expand-file-name "emacs-rag/emacs-rag-search.el" default-directory))
(load-file (expand-file-name "emacs-rag/emacs-rag.el" default-directory))

(emacs-rag-stop-server)
(emacs-rag-start-server)

Quick Start Guide

Using the Transient Menu

The easiest way to use emacs-rag is through the transient menu:

M-x emacs-rag-menu

This opens an organized menu with all commands:

Top Row:

  • Search (v/t/y/h/F): Vector, text, hybrid search, org headings, open files
  • Server (a/p/r/S/l): Start, stop, restart, stats, logs
  • Index (b/f/d/o): Buffer, file, directory, open buffers

Bottom Row:

  • Delete (x/X/R): Buffer, file, database
  • Maintenance (M/B): Rebuild FTS index, rebuild database
  • Debug (D): Debug information

1. Start the Server

M-x emacs-rag-start-server

Or from the transient menu:

M-x emacs-rag-menu
;; Press 'a' to start server

The server will start on http://127.0.0.1:8765 by default.

2. Index Your Documents

Index Current Buffer

M-x emacs-rag-index-buffer

This indexes the current buffer, including any unsaved changes.

Index a Directory

M-x emacs-rag-index-directory
;; Select directory to index

This will recursively index all eligible files (based on emacs-rag-indexed-extensions).

Index a Specific File

M-x emacs-rag-index-file
;; Select file to index

3. Search Your Documents

M-x emacs-rag-search-vector
;; Enter your search query: "machine learning concepts"

Results will be displayed with scores. Select one to navigate directly to that location in the file.

4. Other Useful Commands

Search with Selected Text

All search commands (vector, text, hybrid) automatically use selected region as the query:

;; Select text, then:
M-x emacs-rag-search-vector    ; Semantic search
M-x emacs-rag-search-text      ; Keyword search
M-x emacs-rag-search-hybrid    ; Combined search

Jump to Org Headings

M-x emacs-rag-jump-to-org-heading

Browse all org headings from indexed files with instant navigation.

Search Org Headings Semantically

M-x emacs-rag-search-org-headings

Perform semantic search across org headings. When using Ivy, this provides a dynamic search interface - results update in real-time as you type, continuously re-querying the semantic search engine with your current input.

This is particularly useful for:

  • Finding headings by concept rather than exact wording
  • Exploring related topics across multiple org files
  • Quick navigation when you remember the topic but not the exact heading text

With Ivy: Type continuously and watch results update dynamically Without Ivy: Enter query once, then select from static results

View Statistics

M-x emacs-rag-stats

Shows total indexed chunks and files.

Debug Information

M-x emacs-rag-debug

Displays comprehensive diagnostic information.

reload

;; Load the specific file with full path
(load-file "/Users/jkitchin/Dropbox/emacs/user/emacs-rag-libsql/emacs-rag/emacs-rag-server.el")
(load-file "/Users/jkitchin/Dropbox/emacs/user/emacs-rag-libsql/emacs-rag/emacs-rag-index.el")
(load-file "/Users/jkitchin/Dropbox/emacs/user/emacs-rag-libsql/emacs-rag/emacs-rag-search.el")
(load-file "/Users/jkitchin/Dropbox/emacs/user/emacs-rag-libsql/emacs-rag/emacs-rag-gptel-tools.el")
(load-file "/Users/jkitchin/Dropbox/emacs/user/emacs-rag-libsql/emacs-rag/emacs-rag.el")


(emacs-rag-stop-server)
(emacs-rag-start-server)

(emacs-rag-gptel-enable-tool)

Usage Examples

Example 1: Research Notes

You have a directory of research notes in org-mode:

;; Index your research directory
M-x emacs-rag-index-directory
;; → ~/Documents/research/

;; Search across all notes
M-x emacs-rag-search-vector
;; Query: "neural network optimization techniques"

;; Results show relevant sections from multiple files
;; Select one to jump directly to that content

Example 2: Code Documentation

Search across your project documentation:

;; Add markdown files to indexed types
(setq emacs-rag-indexed-extensions '("org"))

;; Index docs directory
M-x emacs-rag-index-directory
;; → ~/projects/myapp/docs/

;; Search for specific topics
M-x emacs-rag-search-vector
;; Query: "authentication flow"

Example 3: Journal Entries

Search your daily journal by topic:

;; Auto-index enabled - journals update as you save
(setq emacs-rag-auto-index-on-save t)

;; Search across all journal entries
M-x emacs-rag-search-vector
;; Query: "project planning discussions"

;; Find relevant journal entries even if they use different wording

Configuration

Emacs Configuration Variables

Server Settings

(setq emacs-rag-server-host "127.0.0.1")    ; Server hostname
(setq emacs-rag-server-port 8765)           ; Server port
(setq emacs-rag-db-path "~/.emacs-rag/libsql")  ; Database location

Indexing Settings

(setq emacs-rag-indexed-extensions '("org" "txt" "md"))  ; File types
(setq emacs-rag-auto-index-on-save t)       ; Auto-reindex on save

Search Settings

(setq emacs-rag-search-limit 5)             ; Default result count
(setq emacs-rag-search-enable-rerank t)     ; Enable reranking
(setq emacs-rag-result-display-width 80)    ; Result text width

Server Configuration (Environment Variables)

Database

export EMACS_RAG_DB_PATH="$HOME/.emacs-rag/libsql"

Chunking

export EMACS_RAG_CHUNK_SIZE="800"           # Characters per chunk
export EMACS_RAG_CHUNK_OVERLAP="100"        # Overlap between chunks

Models

# Embedding model
export EMACS_RAG_EMBEDDING_MODEL="sentence-transformers/all-MiniLM-L6-v2"

# Alternative: Higher quality but slower
# export EMACS_RAG_EMBEDDING_MODEL="sentence-transformers/all-mpnet-base-v2"

# Reranking model
export EMACS_RAG_RERANK_MODEL="cross-encoder/ms-marco-MiniLM-L-6-v2"

# Enable/disable reranking
export EMACS_RAG_RERANK_ENABLED="true"

# Number of candidates to rerank
export EMACS_RAG_RERANK_TOP_K="20"

Server

export EMACS_RAG_HOST="127.0.0.1"
export EMACS_RAG_PORT="8765"

API Reference

REST API Endpoints

POST /index

Index a file with automatic chunking and embedding.

Request:

{
  "path": "/absolute/path/to/file.org",
  "content": "optional content override",
  "metadata": {
    "author": "John Doe",
    "tags": ["research", "ML"]
  }
}

Response:

{
  "path": "/absolute/path/to/file.org",
  "chunks_indexed": 15
}

GET /search/vector

Semantic similarity search.

Parameters:

  • query (required): Search text
  • limit (optional, default: 5): Max results
  • rerank (optional, default: true): Enable reranking

Response:

{
  "results": [
    {
      "source_path": "/path/to/file.org",
      "chunk_index": 2,
      "line_number": 45,
      "content": "Relevant text content...",
      "score": 0.8534
    }
  ]
}

DELETE /files

Remove all chunks for a file.

Parameters:

  • path (required): Absolute file path

Response:

{
  "path": "/path/to/file.org",
  "deleted": true
}

GET /stats

Database statistics.

Response:

{
  "total_chunks": 1234,
  "total_unique_files": 56,
  "sample_chunk": {...}
}

GET /health

Health check.

Response:

{
  "status": "ok"
}

Emacs Commands

Server Management

CommandDescription
emacs-rag-start-serverStart the RAG server
emacs-rag-stop-serverStop the RAG server
emacs-rag-restart-serverRestart the RAG server
emacs-rag-show-server-bufferShow server log buffer

Indexing

CommandDescription
emacs-rag-index-fileIndex a specific file
emacs-rag-index-bufferIndex current buffer
emacs-rag-index-directoryRecursively index directory
emacs-rag-reindex-all-open-buffersReindex all open eligible buffers
emacs-rag-delete-fileRemove file from index
emacs-rag-delete-bufferRemove current buffer from index

Search

CommandDescription
emacs-rag-search-vectorSemantic vector search (uses region)
emacs-rag-search-textFull-text FTS5 search (uses region)
emacs-rag-search-hybridHybrid vector + text search (uses region)
emacs-rag-search-org-headingsSemantic search of org headings (dynamic with Ivy)
emacs-rag-jump-to-org-headingNavigate to any org heading
emacs-rag-open-indexed-fileBrowse and open indexed files
emacs-rag-statsShow database statistics

Utilities

CommandDescription
emacs-rag-menuOpen transient menu
emacs-rag-debugShow debug information
emacs-rag-quick-startShow quick start guide
emacs-rag-delete-databaseDelete entire database
emacs-rag-rebuild-databaseRebuild database with new schema
=emacs-rag-rebuild-fts-index=Rebuild FTS5 index from documents

Advanced Usage

Custom Metadata

Add custom metadata when indexing:

(emacs-rag-index-file
 "~/notes/research.org"
 '((author . "John Doe")
   (project . "ML Research")
   (tags . ("neural-networks" "optimization"))))

Programmatic Search

(let* ((results (emacs-rag--request
                 "GET" "/search/vector" nil
                 '((query . "machine learning")
                   (limit . 10)
                   (rerank . "true"))))
       (top-result (car (alist-get 'results results))))
  ;; Process results programmatically
  (message "Top result: %s (score: %.3f)"
           (alist-get 'source_path top-result)
           (alist-get 'score top-result)))

Batch Indexing with Progress

(defun my-index-project ()
  "Index all org files in current project."
  (interactive)
  (when-let ((project-root (project-root (project-current))))
    (message "Indexing project: %s" project-root)
    (emacs-rag-index-directory project-root)))

LLM Integration with gptel

The emacs-rag-gptel-tools module provides function calling tools that allow LLMs (via gptel) to search your indexed documents and retrieve relevant information during AI interactions.

Setup

First, ensure you have gptel installed with tool support:

;; Load the gptel tools module
(require 'emacs-rag-gptel-tools)

;; Enable the RAG search tool
(emacs-rag-gptel-enable-tool)

Usage

Once enabled, when you interact with an LLM through gptel, it can automatically call the rag_search tool to retrieve relevant information from your indexed documents:

;; Example: Ask the LLM a question about your documents
;; The LLM will automatically use rag_search if it needs information

M-x gptel-send

Prompt: "What did I write about machine learning optimization in my notes?it m"

;; The LLM will:
;; 1. Call rag_search with query "machine learning optimization"
;; 2. Receive the full text of the most relevant file
;; 3. Use that information to answer your question

Available Tool

=rag_search=: Searches through indexed documents using semantic vector search and returns the full text of the top matching file.

Parameters:

  • query (string): The search query to find relevant documents

The tool automatically handles:

  • Server availability checking
  • Vector search with reranking enabled
  • Retrieving the full file content
  • Returning formatted results with relevance scores

Disabling the Tool

To disable the RAG search tool:

(emacs-rag-gptel-disable-tool)

Different Embedding Models

For better quality (but slower):

export EMACS_RAG_EMBEDDING_MODEL="sentence-transformers/all-mpnet-base-v2"
emacs-rag-server serve

For multilingual support:

export EMACS_RAG_EMBEDDING_MODEL="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
emacs-rag-server serve

Troubleshooting

Server Won’t Start

;; Check server buffer for errors
M-x emacs-rag-show-server-buffer

;; Check debug info
M-x emacs-rag-debug

;; Verify Python installation
M-x shell-command RET python3 --version

No Search Results

  • Verify files are indexed: M-x emacs-rag-stats
  • Check if server is running: M-x emacs-rag-debug
  • Try disabling reranking temporarily
  • Increase search limit with prefix argument: C-u 10 M-x emacs-rag-search-vector

Poor Search Quality

  • Enable reranking: (setq emacs-rag-search-enable-rerank t)
  • Increase reranking pool: export EMACS_RAG_RERANK_TOP_K=30
  • Try a different embedding model (see Advanced Usage)
  • Adjust chunk size: export EMACS_RAG_CHUNK_SIZE=1000

Indexing Fails

  • Check file permissions
  • Verify file encoding (UTF-8 recommended)
  • Check available disk space
  • Review server logs: M-x emacs-rag-show-server-buffer

High Memory Usage

  • Use a smaller embedding model
  • Reduce chunk overlap: export EMACS_RAG_CHUNK_OVERLAP=50
  • Clear old indexes: M-x emacs-rag-delete-database

Performance Considerations

Indexing Speed

  • Chunk size: Larger chunks = fewer embeddings = faster indexing
  • Batch size: Currently fixed at 8 documents per batch
  • Model: all-MiniLM-L6-v2 is the fastest default model

Search Speed

  • Vector search: Very fast (milliseconds)
  • Reranking: Slower but only applied to top-K candidates
  • Adjust rerank_top_k: Lower values = faster search, potentially less accurate

Storage

  • Embeddings: 384 floats × 4 bytes = ~1.5KB per chunk
  • Text: Depends on chunk size (default 800 chars ≈ 800 bytes)
  • Typical: ~2-3KB per chunk including metadata

Development

Project Structure

emacs-rag-libsql/
├── emacs-rag/                    # Emacs Lisp package
│   ├── emacs-rag.el              # Main entry point + menu
│   ├── emacs-rag-server.el       # Server management
│   ├── emacs-rag-index.el        # Indexing commands
│   └── emacs-rag-search.el       # Search interface
├── emacs-rag-server/             # Python FastAPI server
│   ├── src/emacs_rag_server/
│   │   ├── main.py               # FastAPI app
│   │   ├── cli.py                # CLI interface
│   │   ├── api/routes.py         # API endpoints
│   │   ├── models/               # Database, embeddings, schemas
│   │   ├── services/             # Business logic
│   │   └── utils/                # Utilities
│   ├── pyproject.toml
│   └── README.org
├── software-design.org           # Design documentation
└── readme.org                    # This file

Running Tests

cd emacs-rag-server
uv sync --dev
uv run pytest

Development Mode

Start server with auto-reload:

emacs-rag-server serve --reload

Interactive API Documentation

When the server is running:

  • Swagger UI: http://127.0.0.1:8765/docs
  • ReDoc: http://127.0.0.1:8765/redoc

Comparison with Other Tools

vs. Traditional Grep/Ripgrep

Featureemacs-rag-libsqlgrep/ripgrep
Search TypeSemanticKeyword/Regex
Finds Concepts
SpeedFast (indexed)Very Fast
Setup RequiredYesNo
Memory UsageModerateLow
RankingML-basedNone

vs. Org-roam

Featureemacs-rag-libsqlorg-roam
Search TypeSemantic full-textLinks + Tags
Structure RequiredNoYes (IDs, links)
Content Search✓ AdvancedBasic
Relationship Mapping
Backlinks

vs. Deft

Featureemacs-rag-libsqlDeft
Search TypeSemantic vectorKeyword
Relevance RankingML-basedFrequency
File NavigationLine-levelFile-level
PerformanceIndexed (fast)Live search

Future Enhancements

Potential features for future development:

  • [ ] PDF/DOCX indexing (via docling)
  • [ ] Multiple collection support
  • [ ] Project-scoped search
  • [ ] org-db integration
  • [ ] Metadata-based filtering in search
  • [ ] Incremental indexing (detect changes)
  • [ ] Search result caching
  • [ ] Export/import database
  • [ ] Remote server support
  • [ ] Date-based filtering
  • [ ] Duplicate detection
  • [X] Integration with GPT for RAG (via gptel tools)

License

This project is licensed under the MIT License. See the LICENSE file for details.

Copyright (c) 2025 John Kitchin

Contributing

Contributions are welcome! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests if applicable
  5. Submit a pull request

Support

For issues, questions, or suggestions:

  • Check the troubleshooting section above
  • Review M-x emacs-rag-debug output
  • Check server logs with M-x emacs-rag-show-server-buffer
  • File an issue on the project repository

Acknowledgments

This project uses:

  • FastAPI - Modern web framework for Python
  • LibSQL - SQLite fork with vector support
  • Sentence Transformers - State-of-the-art text embeddings
  • Transient - Emacs transient command interface
  • Ivy - Completion framework for Emacs (optional)

References

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published