Repository navigation
1.5.0
What's new
- Long-context search without OOMs — scoring batch and chunk sizes are now computed automatically from a per-device memory budget, in both the approximate scoring and the exact rerank stages.
search(batch_size=...)takes'auto'(default, planned fromsearch_memory_fractionof the free VRAM at query time) or an int to force fixed-size chunks. Chunking only kicks in when a query needs it, so short queries keep a single-chunk fast path, and a CUDA OOM retries the stage with the budget halved. - Adaptive index placement —
index_gpu_memory='auto' | 'low' | 'medium' | 'high'replaceslow_memory.'high'keeps codes and residuals on GPU,'medium'keeps codes on GPU and residuals (the larger tensor) on CPU,'low'keeps both on CPU, and'auto'(default) picks the highest tier that keeps device occupancy withinindex_memory_fraction(0.7).low_memoryis still accepted, as a keyword or positionally, but is ignored and raises aDeprecationWarning. - Packed query tensors — queries are passed as one packed 2D tensor plus per-query lengths instead of a padded 3D tensor, so there is no padding waste on GPU and only a single host-to-device transfer. The public Python API is unchanged.
- In-place MaxSim masking — the mask fill is applied in place on the freshly materialized score tensor, avoiding an extra full-size GPU allocation.
- No GPU load in
create()— per-device loads are deferred to the first search, since the index object built during creation was discarded anyway. - Per-query search errors propagate — a failing query now raises instead of being silently returned as an empty result.
See PR #46 for details.