v2.0.0

[2.0.0] - 2025-12-23

Breaking Changes

• Backend Migration: sqlite-vec → usearch HNSW

  • Vector search now uses usearch's high-performance HNSW algorithm
  • 10-100x faster similarity search for large collections
  • Vector data stored in separate .usearch files per collection (e.g., mydb.db.default.usearch)
  • SQLite still stores metadata, text, and FTS5 index

• Removed DistanceStrategy.L1 - Manhattan distance not supported by usearch

• Storage Format Change

  • Embeddings now stored in both usearch index AND SQLite (for MMR support)
  • Existing sqlite-vec databases will auto-migrate on first open
  • Migration is one-way; backup before upgrading

Added

• usearch_index.py - New UsearchIndex wrapper class:

  • Thread-safe HNSW index operations (lock on writes, lock-free reads)
  • Automatic persistence to .usearch files
  • Upsert support (removes existing keys before add)
  • BIT quantization using Hamming metric with bit packing
  • Configurable HNSW parameters (connectivity, expansion_add, expansion_search)

• Proper MMR Implementation - Max Marginal Relevance now computes actual pairwise similarity between candidates and selected documents using stored embeddings

• Embedding Storage in SQLite - Embeddings stored as BLOB for:

  • Accurate MMR diversity computation
  • Future index rebuild from SQLite backup
  • Schema auto-migrates existing tables

• VectorCollection.rebuild_index() - Reconstruct usearch HNSW index from SQLite embeddings:

  • Useful for index corruption recovery
  • Tune HNSW parameters (connectivity, expansion_add, expansion_search)
  • Reclaim space after many deletions

• VectorDB.check_migration(path) - Dry-run migration check:

  • Reports which collections need migration
  • Shows total vector count and estimated storage
  • Provides detailed rollback instructions

• Adaptive Search - Automatically optimizes search strategy based on collection size:

  • Collections < 10k vectors use brute-force (exact=True) for perfect recall
  • Collections ≥ 10k vectors use HNSW for faster approximate search
  • Threshold configurable via constants.USEARCH_BRUTEFORCE_THRESHOLD

• exact parameter - Force search mode in similarity_search():

  • None (default): adaptive based on collection size
  • True: force brute-force for perfect recall
  • False: force HNSW approximate search

• Quantization.FLOAT16 - Half-precision floating point:

  • 2x memory savings compared to FLOAT32
  • 1.5x faster search with minimal precision loss
  • Ideal for embeddings where full precision isn't needed

• threads parameter - Parallel execution control:

  • Added to add_texts() and similarity_search()
  • 0 (default): auto-detect optimal thread count
  • Explicit value: control parallelism for batch operations

• Auto Memory-Mapping - Large indexes automatically use memory-mapped mode:

  • Indexes >100k vectors use view=True for instant startup
  • Lower memory footprint for large collections
  • Transparent upgrade to writable mode on add operations
  • Configurable via constants.USEARCH_MMAP_THRESHOLD

• similarity_search_batch() - Multi-query batch search:

  • ~10x throughput for batch query workloads
  • Uses usearch's native batch search under the hood
  • Same parameters as similarity_search() but accepts list of queries

• examples/backend_benchmark.py - Benchmark script comparing usearch vs brute-force:

  • Measures speedup, recall, and storage efficiency
  • Supports all quantization levels
  • Validates 10-100x performance claims

Changed

• Dependencies: Replaced sqlite-vec>=0.1.6 with usearch>=2.12
• CatalogManager: Removed vec0 virtual table operations, added embedding column
• SearchEngine: Rewrote to use UsearchIndex for all vector operations
• VectorCollection: Creates usearch index at {db_path}.{collection}.usearch

Migration Notes

1. Backup your database before upgrading
2. On first open, existing sqlite-vec data will be migrated automatically
3. New .usearch files will be created alongside your .db file
4. The legacy sqlite-vec table is dropped after successful migration