Repository navigation
The index is stored once (#53, #54)
Codes and residuals no longer exist twice on disk. The merged files are now the single durable copy; per-shard files are transient staging that the next merge folds in and unlinks. Mutable indexes take ~half the disk (−46% measured at 50k docs and under sustained update/delete churn).
update()/delete()/ search results are bit-identical to 1.6.0 operating on the same index (validated per-query on SciFact: 300 queries × top-100, zero ranking or score differences).- Existing indexes convert automatically on first load; search results are identical before and after.
- Downgrade-safe: an index touched by 1.7.0 opens read-only under older versions (instead of appearing empty), and the older version's
unfreeze()fully converts it back to the double-copy layout. freeze()/unfreeze()are now instant metadata toggles.- Crash-recovery by ordering: staging files are unlinked only after the manifest acknowledging them is durably written; a lost manifest is rebuilt from the doclens files.
- The merged npy header has a fixed length, removing the silent full-file rewrite when the row count crossed a digit boundary; loads that have nothing to merge no longer write anything.
- Opt out with
FAST_PLAID_KEEP_SHARDS=1.
Validated with mLateOn on BEIR (SciFact / NFCorpus / ArguAna / SCIDOCS): paper-matching nDCG@10 through both create() and incremental update(), plus 25-cycle update/random-delete churn with zero id drift and unchanged VRAM/RSS peaks vs 1.6.0.
Known issue spotted during validation (pre-existing, unchanged by this release): integer batch_size bypasses the adaptive memory planner at search time — see #55. Prefer batch_size="auto".