Repository navigation
Is a Vamana sub-index for IVF wanted in Lance? #10590
wombatu-kun
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I opened #10318, which adds Vamana as a third IVF sub-index next to FLAT and HNSW, and I would like a direction before going further: is this something Lance wants in-tree?
On four 1M-scale datasets (SIFT, GloVe-200, Cohere, GIST), at the same recall,
IVF_VAMANA_SQanswers 1.05-1.82x faster than the faster ofIVF_HNSW_SQandIVF_RQwith one query in flight, and serves 1.16-2.78x more queries per second with 24 in flight. The costs: builds take 1.15-1.63x as long asIVF_HNSW_SQand 10-34x as long asIVF_RQ, and the index is 9-30% smaller thanIVF_HNSW_SQbut 1.7-2.4x the size ofIVF_RQ. Measured on a 12-core laptop with Lance 14.0.0-beta.13; full tables, settings and known limitations are in the PR.The change is large: about 21,400 new lines, 13,500 of them tests, and about 770 lines in existing files. It also needs the format vote, for a new
VectorIndexDetailsfield and the graph files. Released builds hide these indices instead of failing on them (checked with pylance 13.0.0).So the question is which of these you would prefer:
I also could not find why #1994 removed the earlier DiskANN index. If the reason still applies, that alone may decide this.
@BubbleCal @westonpace this touches your areas (vector search and #5231), so I would value your view.
All reactions