Every modality, one search. Local first.
Standard RAG only works on text. Drop a PDF with charts, a photo of a whiteboard, or a video recording — and your AI agent goes blind. RecallForge gives agents eyes and ears over your local filesystem. Text, images, documents, and video all live in one unified search space, and nothing ever leaves your machine.
You: "What did the whiteboard look like in our last meeting?"
Claude: (Searches your local
~/Documents, finds a photo of a whiteboard from an iPhone, reads the handwriting via Qwen3-VL, and surfaces the image with context.)
You: "Find the architecture diagram from that PDF I downloaded last week."
Claude: (Indexes the PDF, matches your query against extracted text and embedded figures, returns the relevant page.)
You: (Drops an image of a circuit board) "Find my notes related to this."
Claude: (Reverse image-to-text search across your indexed notes. Returns matching documents.)
One query. Any modality. All local.
| Capability | RecallForge | Chroma | Mem0 | Qdrant | Weaviate |
|---|---|---|---|---|---|
| Cross-modal search | ✅ Native | ✅ OpenCLIP | ❌ Text only | ❌ | ✅ CLIP modules |
| Video support [Beta] | ✅ | ❌ | ❌ | ❌ | ❌ |
| Document ingest (PDF/DOCX/PPTX) | ✅ | ❌ | ❌ | ❌ | ❌ |
| Built-in reranking | ✅ Multimodal | ❌ | ❌ | ✅ ColBERT | ✅ Modules |
| MCP-native | ✅ 20 tools | ❌ | ❌ | ❌ | ❌ |
| 100% local | ✅ | ✅ | ✅ | ✅ Docker | |
| Apple Silicon optimized | ✅ MLX 4-bit | ❌ | ❌ | ❌ | ❌ |
| Cloud option | ❌ | ✅ | ✅ | ✅ | ✅ |
| JS/TS SDK | ❌ | ✅ | ✅ | ✅ | ✅ |
Use RecallForge when: You need multimodal memory for AI agents that runs entirely on your machine, especially on Apple Silicon. One search across text, images, documents, and video.
Use something else when: You need cloud hosting, massive scale (millions+ vectors), or a JS/TS-first ecosystem.
4 modalities (text, images, documents, video) unified in a single MLX-optimized local vector space. Sub-60ms search latency in embed mode. Under 400MB resident memory.
Each stage of the pipeline improves retrieval quality. The reranker is the quality peak.
| Stage | R@1 | R@5 | R@10 | MRR | p50 |
|---|---|---|---|---|---|
| Vector-only | 65.2% | 65.2% | 67.4% | 67.3% | 20ms |
| BM25-only | 57.6% | 57.6% | 93.5% | 64.4% | 17ms |
| Vector + BM25 (RRF) | 69.6% | 88.0% | 90.2% | 77.5% | 100ms |
| + Reranker (hybrid mode) | 85.9% | 92.4% | 97.8% | 89.2% | 3.8s |
The reranker delivers +20.7% R@1 over RRF fusion and pushes R@10 to 97.8%. Embed mode gives you 20ms searches for speed-sensitive workloads. Hybrid mode gives you 85.9% R@1 when quality matters.
Benchmark categories: text_only (30 queries), image_only (30 queries), long_query (12 queries), typo_query (20 queries). See benchmarks/results/pipeline_ablation_modality_results.json for full breakdown.
For release validation, use benchmarks/cross_modal_ablation.py. It now checkpoints JSON output as it runs, so long MLX benchmark sessions still leave behind a partial artifact if interrupted.
| Metric | MLX 4-bit | PyTorch fp16 |
|---|---|---|
| Warm search p50 (embed) | 53ms | 599ms |
| Warm search p95 (embed) | 55ms | — |
| Cold start | 7.6s | ~20s |
| Peak RSS (embed) | 329MB* | ~4GB |
| Peak RSS (hybrid) | ~1.5GB* | ~5GB |
| Text indexing | 5.0 docs/sec | — |
*MLX maps model weights lazily via memory-mapped files. RSS reflects resident pages, not full model size (~1.7GB embedder + ~1.7GB reranker on disk). Actual memory pressure is low.
For transparency: raw embedding quality on the standard COCO benchmark (1,000 images, no BM25/reranking/expansion). These numbers reflect the Qwen3-VL-2B embedder alone, not the full pipeline.
| Direction | R@1 | R@5 | R@10 |
|---|---|---|---|
| Text → Image | 24.5% | 42.3% | 49.9% |
| Image → Text | 34.3% | 42.0% | 44.1% |
Qwen3-VL is a generative VLM, not a contrastive model like CLIP. The pipeline ablation above shows how BM25 fusion and reranking compensate for this.
pip install recallforge[mlx] # Apple Silicon (recommended, 4-bit quantization)
pip install "recallforge[mlx,server]" # Apple Silicon + HTTP/SSE server
pip install recallforge[cuda] # NVIDIA GPU
pip install recallforge[torch] # CPU / other PyTorch targets
pip install recallforge[docs] # add richer PDF extraction (optional)Note:
pip install recallforgeinstalls the core without a backend. You need at least one of[mlx],[cuda], or[torch]to run inference. Add[server]only when you want HTTP/SSE transport (recallforge serve --http).
From source:
git clone https://github.com/brianmeyer/recallforge.git
cd recallforge
pip install -e ".[mlx]"- Python 3.12 or 3.13 required (3.14 not yet supported, pending pyarrow wheel)
- Disk: ~2-5GB free for model downloads on first run
- RAM (MLX 4-bit): ~1.7GB (
embed) to ~3.4GB (hybrid) ffmpegrecommended for video indexing/search- First run downloads models automatically and may take a few minutes
RecallForge is designed as a Model Context Protocol server for AI agents. Configure in Claude Desktop (or any MCP-compatible agent host):
{
"mcpServers": {
"recallforge": {
"command": "recallforge",
"args": ["serve", "--mode", "hybrid"]
}
}
}Run manually (stdio):
recallforge serve --mode embed --backend mlx --quantize 4bitRun over HTTP/SSE:
recallforge serve --http --host 127.0.0.1 --port 7433 --mode embedRecallForge now exposes 20 MCP tools across search, ingest, memory, collection admin, and runtime config. HTTP/SSE mode also exposes /health, /sse, and /messages/.
See docs/mcp-tools.md for the full tool reference.
| Mode | Models loaded | Memory (MLX 4-bit) | Quality | Best for |
|---|---|---|---|---|
embed |
Embedder | ~1.7GB | Good | Memory-constrained, fast searches |
hybrid |
Embedder + Reranker | ~3.4GB | Best | Maximum retrieval quality |
Video [Beta] note: Video support requires
ffmpeg. The torch backend video path has a known upstream issue (see QwenLM/Qwen3.5#58).
RecallForge encodes text, images, and video frames into the same 2048-dimensional vector space using Qwen3-VL. This means "find notes about this diagram" works whether the diagram is text, an image, or a frame from a video. A 3-stage pipeline handles the rest:
graph TD
subgraph Local Filesystem
Docs[📄 Documents]
Imgs[🖼️ Images]
Vids[🎬 Video]
end
subgraph RecallForge Ingest
Docs --> TxtExt[Text Extractor]
Imgs --> VLM[Qwen3-VL Encoder]
Vids --> Frame[Frame & Audio Extractor]
Frame --> VLM
TxtExt --> VLM
end
subgraph LanceDB Storage
VLM -->|2048-dim Vectors| VecDB[(Vector Space)]
TxtExt -->|Text/Transcripts| FTS[(Tantivy FTS)]
end
subgraph MCP Search Pipeline
Query[Agent Query] --> BM25[BM25 Text Search]
Query --> Dense[Vector Similarity Search]
BM25 --> RRF[RRF Fusion]
Dense --> RRF
RRF --> Rerank[Cross-Encoder Reranker]
Rerank --> Output[Final Context to Agent]
end
Pipeline: BM25 probe → Parallel BM25 + Vector → RRF fusion → Reranking (hybrid mode) → Score blending
# Index anything
recallforge index ./photos ./docs
recallforge index ~/Movies/demo.mp4
recallforge index ~/Documents/roadmap.pptx
# Search any modality
recallforge search "whiteboard diagram from last meeting"
recallforge search --image ./photos/whiteboard.png
recallforge search --video ~/Movies/demo.mp4
# Watch a folder for changes (auto-index)
recallforge watch start ~/Documents --collection docs
recallforge watch list
recallforge watch stop ~/Documents
# Status
recallforge statusRecallForge auto-detects MLX on Apple Silicon, PyTorch elsewhere.
from recallforge import get_backend, get_storage
from recallforge.search import HybridSearcher
backend = get_backend()
storage = get_storage()
backend.warm_up()
# Index
storage.index_document(
path="notes.md",
text="My notes about AI...",
collection="my_docs",
model="Qwen3-VL-Embedding-2B",
embed_func=backend.embed_text,
)
# Search
searcher = HybridSearcher(backend=backend, storage=storage, limit=10)
results = searcher.search("artificial intelligence")
for r in results:
print(f"[{r.score:.3f}] {r.title}")| Variable | Default | Description |
|---|---|---|
RECALLFORGE_BACKEND |
auto |
auto, mlx, torch |
RECALLFORGE_MODE |
hybrid |
embed, hybrid |
RECALLFORGE_MLX_QUANTIZE |
4bit |
4bit, bf16 |
RECALLFORGE_STORE_PATH |
~/.recallforge |
Storage directory |
Full reference (including advanced tuning and server/storage internals):
docs/ENV_VARS.md
src/recallforge/
├── backends/
│ ├── mlx_backend.py # MLX 4-bit/bf16 (Apple Silicon)
│ └── torch_backend.py # PyTorch (CUDA/MPS/CPU)
├── storage/
│ └── lancedb_backend.py # LanceDB + Tantivy FTS
├── cache.py # LRU embedding cache
├── search.py # Hybrid search pipeline (BM25 + vector + RRF)
├── server.py # MCP server (20 tools, stdio + HTTP/SSE)
├── documents.py # PDF/DOCX/PPTX extraction
├── video.py # Frame/transcript extraction
├── watch_folder.py # Folder monitoring with dedup
└── cli.py # CLI interface
pytest tests/ -m "not live" # Unit tests (no model download needed)
pytest tests/ -m live -v # Integration tests (requires models)CI in .github/workflows/ci.yml runs the test matrix, builds distributions, runs twine check, smoke-tests wheel installation, and smoke-tests the HTTP server extra from the built wheel. Tagged pushes matching v* trigger .github/workflows/publish.yml, which publishes to PyPI with trusted publishing.
Before tagging a release, run the repo test suite plus the install/CLI UAT scripts, and if you are on a capable host, run the live integration slice and expanded benchmark. The full checklist lives in docs/RELEASE.md.
See CONTRIBUTING.md for full development guidelines.
RecallForge is inspired by QMD by Tobi. QMD pioneered the multi-stage retrieval pipeline (embedding, reranking). RecallForge extends this pattern to vision-language with cross-modal retrieval and multi-backend support.
MIT License