Simple by default, powerful when needed
A local-first TypeScript retrieval engine for semantic search over static documents. Built to be simple to use, lightweight, and hackable with zero external run-time dependencies.
- Quick Start
- Features
- How It Works
- Supported Models
- Documentation
- MCP Server Integration
- Development
- Contributing
- License
npm install -g rag-lite-ts
# Ingest documents
raglite ingest ./docs/
# Search your documents
raglite search "machine learning concepts"
# Get more results with reranking
raglite search "API documentation" --top-k 10 --rerank
# Use higher quality model (auto-rebuilds if needed)
raglite ingest ./docs/ --model Xenova/all-mpnet-base-v2 --rebuild-if-needed
# Search automatically uses the correct model
raglite search "complex query"
import { SearchEngine, IngestionPipeline } from 'rag-lite-ts';
// Initialize and ingest documents
const ingestion = new IngestionPipeline('./db.sqlite', './vector-index.bin');
await ingestion.ingestDirectory('./docs/');
// Search your documents
const search = new SearchEngine('./vector-index.bin', './db.sqlite');
const results = await search.search('machine learning', { top_k: 10 });
import { SearchEngine, IngestionPipeline } from 'rag-lite-ts';
// Custom model configuration
const search = new SearchEngine('./vector-index.bin', './db.sqlite', {
embeddingModel: 'Xenova/all-mpnet-base-v2',
enableReranking: true,
topK: 15
});
// Ingestion with custom settings
const ingestion = new IngestionPipeline('./db.sqlite', './vector-index.bin', {
embeddingModel: 'Xenova/all-mpnet-base-v2',
chunkSize: 400,
chunkOverlap: 80
});
β Complete CLI Reference | API Documentation
- π Simple: Get started with just
new SearchEngine()
- no complex setup required - π Local-first: All processing happens offline on your machine
- π Fast: Sub-100ms queries for typical document collections
- π Semantic: Uses embeddings for meaning-based search, not just keywords
- π οΈ Flexible: Simple constructors for basic use, advanced options when you need them
- π¦ Complete: CLI, programmatic API, and MCP server in one package
- π― TypeScript: Full type safety with modern ESM architecture
- π§ Smart: Automatic model management and compatibility checking
RAG-lite TS follows a simple pipeline:
- Document Ingestion: Reads
.md
,.txt
,.mdx
,.pdf
, and.docx
files - Preprocessing: Cleans content (JSX components, Mermaid diagrams, code blocks)
- Semantic Chunking: Splits documents at natural boundaries with token limits
- Embedding Generation: Uses transformers.js models for semantic vectors
- Vector Storage: Fast similarity search with hnswlib-wasm
- Metadata Storage: SQLite for document info and model compatibility
- Search: Embeds queries and finds similar chunks using cosine similarity
- Reranking (optional): Cross-encoder models for improved relevance
Documents β Preprocessor β Chunker β Embedder β Vector Index
β
Query β Embedder β Vector Search β SQLite Lookup β Results
β Document Preprocessing Guide | Model Management Details
RAG-lite TS supports multiple embedding models with automatic optimization:
Model | Dimensions | Speed | Use Case |
---|---|---|---|
sentence-transformers/all-MiniLM-L6-v2 |
384 | Fast | General purpose (default) |
Xenova/all-mpnet-base-v2 |
768 | Slower | Higher quality, complex queries |
Model Features:
- Automatic downloads: Models cached locally on first use
- Smart compatibility: Detects model changes and prompts rebuilds
- Offline support: Pre-download for offline environments
- Reranking: Optional cross-encoder models for better relevance
β Complete Model Guide | Performance Benchmarks
- CLI Reference - Installation and basic usage
- API Reference - Simple constructors and programmatic usage
- Configuration Guide - Custom settings and options
- Model Selection Guide - Choose the right model for your needs
- Path Storage Strategies - Document path management
- Document Preprocessing - Content processing options
- Troubleshooting Guide - Common issues and solutions
- Embedding Models Comparison - Detailed benchmarks
- Documentation Hub - Complete documentation index
User Type | Start Here | Next Steps |
---|---|---|
New Users | CLI Reference | API Reference |
App Developers | API Reference | Configuration Guide |
Performance Optimizers | Model Guide | Performance Benchmarks |
Production Deployers | Configuration Guide | Path Strategies |
Troubleshooters | Troubleshooting Guide | Preprocessing Guide |
RAG-lite TS includes a Model Context Protocol (MCP) server for integration with AI agents.
# Start MCP server
raglite-mcp
MCP Configuration:
{
"mcpServers": {
"rag-lite": {
"command": "raglite-mcp",
"args": []
}
}
}
Available Tools: search_documents
, ingest_documents
, rebuild_index
, get_stats
β Complete MCP Integration Guide
# Clone and setup
git clone https://github.com/your-username/rag-lite-ts.git
cd rag-lite-ts
npm install
# Build and link for development
npm run build
npm link # Makes raglite/raglite-mcp available globally
# Run tests
npm test
npm run test:integration
src/
βββ index.ts # Main exports and factory functions
βββ search.ts # Public SearchEngine API
βββ ingestion.ts # Public IngestionPipeline API
βββ core/ # Model-agnostic core layer
β βββ search.ts # Core search engine
β βββ ingestion.ts # Core ingestion pipeline
β βββ db.ts # SQLite operations
β βββ config.ts # Configuration system
β βββ types.ts # Core type definitions
βββ factories/ # Factory functions for easy setup
β βββ text-factory.ts # Text-specific factories
βββ text/ # Text-specific implementations
β βββ embedder.ts # Text embedding generation
β βββ reranker.ts # Text reranking
β βββ tokenizer.ts # Text tokenization
βββ cli.ts # CLI interface
βββ mcp-server.ts # MCP server
βββ preprocessors/ # Content type processors
dist/ # Compiled output
Simple by default, powerful when needed:
- β Simple constructors work immediately with sensible defaults
- β Configuration options available when you need customization
- β Advanced patterns available for complex use cases
- β Clean architecture with minimal dependencies
- β No ORMs or heavy frameworks - just TypeScript and SQLite
- β Extensible design for future capabilities
This approach ensures that basic usage is effortless while providing the flexibility needed for advanced scenarios.
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests for new functionality
- Ensure all tests pass
- Submit a pull request
We welcome contributions that maintain our clean architecture principles while enhancing functionality and developer experience.
MIT License - see LICENSE file for details.
- transformers.js - Client-side ML models
- hnswlib - Fast approximate nearest neighbor search