Skip to content

1.5.0

Choose a tag to compare

@raphaelsty raphaelsty released this 04 Aug 13:10
· 16 commits to main since this release

What's new

  • Long-context search without OOMs — scoring batch and chunk sizes are now computed automatically from a per-device memory budget, in both the approximate scoring and the exact rerank stages. search(batch_size=...) takes 'auto' (default, planned from search_memory_fraction of the free VRAM at query time) or an int to force fixed-size chunks. Chunking only kicks in when a query needs it, so short queries keep a single-chunk fast path, and a CUDA OOM retries the stage with the budget halved.
  • Adaptive index placement — index_gpu_memory='auto' | 'low' | 'medium' | 'high' replaces low_memory. 'high' keeps codes and residuals on GPU, 'medium' keeps codes on GPU and residuals (the larger tensor) on CPU, 'low' keeps both on CPU, and 'auto' (default) picks the highest tier that keeps device occupancy within index_memory_fraction (0.7). low_memory is still accepted, as a keyword or positionally, but is ignored and raises a DeprecationWarning.
  • Packed query tensors — queries are passed as one packed 2D tensor plus per-query lengths instead of a padded 3D tensor, so there is no padding waste on GPU and only a single host-to-device transfer. The public Python API is unchanged.
  • In-place MaxSim masking — the mask fill is applied in place on the freshly materialized score tensor, avoiding an extra full-size GPU allocation.
  • No GPU load in create() — per-device loads are deferred to the first search, since the index object built during creation was discarded anyway.
  • Per-query search errors propagate — a failing query now raises instead of being silently returned as an empty result.

See PR #46 for details.