feat: define Flat index as a scan over VectorStorage #2380

chebbyChefNEQ · 2024-05-22T23:46:35Z

Stack from ghstack (oldest at bottom):

closes feat: define Flat index as a scan over VectorStorage #2380
closes feat: add sub index trait #2378
closes chore: move VectorStorage and DistCalculator to v3 module #2377
closes feat: define IVFShuffler trait #2376

[ghstack-poisoned]

ghstack-source-id: 5d8adb1c67e07d251cc9c617dd144b7f9660242d Pull Request resolved: #2380

codecov-commenter · 2024-05-23T00:02:53Z

Codecov Report

Attention: Patch coverage is 0% with 48 lines in your changes are missing coverage. Please review.

Please upload report for BASE (gh/chebbyChefNEQ/5/base@29d23c9). Learn more about missing BASE report.

Files	Patch %	Lines
rust/lance-index/src/vector/v3/subindex.rs	0.00%	48 Missing ⚠️

Additional details and impacted files

@@                    Coverage Diff                     @@
##             gh/chebbyChefNEQ/5/base    #2380   +/-   ##
==========================================================
  Coverage                           ?   79.96%           
==========================================================
  Files                              ?      200           
  Lines                              ?    54519           
  Branches                           ?    54519           
==========================================================
  Hits                               ?    43597           
  Misses                             ?     8400           
  Partials                           ?     2522

Flag	Coverage Δ
unittests	`79.96% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

eddyxu · 2024-05-23T01:24:09Z

rust/lance-index/src/vector/v3/subindex.rs

+impl<T: Num, Store: VectorStore> IvfSubIndex<T, Store> for FlatIndex {
+    type QueryParams = ();
+
+    fn index_name(&self) -> &str {


flat is not IVF_FLAT?

eddyxu · 2024-05-23T01:58:20Z

rust/lance-index/src/vector/v3/subindex.rs

@@ -60,3 +69,71 @@ pub trait IvfSubIndex<T: Num, Store: VectorStore>: Send + Sync + Sized {
        self.to_array()
    }
 }
+
+struct FlatIndex {}


pub?
Please add comments?

rust/lance-index/src/vector/v3/subindex.rs

[ghstack-poisoned]

ghstack-source-id: 64bee5f06dc4868e11e9754f73a2f7d1b68dbe41 Pull Request resolved: #2380

eddyxu

Pending addressing comments.

[ghstack-poisoned]

ghstack-source-id: dcbf9822d8e276436ed51685967fad3773be0932 Pull Request resolved: #2380

[ghstack-poisoned]

ghstack-source-id: 8c523fd8abb4f69352fa6998a25090e8c2ea1e6c Pull Request resolved: #2380

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [lance](https://togithub.com/lancedb/lance) | dependencies | minor | `0.10.16` -> `0.12.0` | --- ### Release Notes <details> <summary>lancedb/lance (lance)</summary> ### [`v0.12.1`](https://togithub.com/lancedb/lance/releases/tag/v0.12.1) [Compare Source](https://togithub.com/lancedb/lance/compare/v0.12.0...v0.12.1)  #### What's Changed ##### Bug Fixes 🐛 - fix: incorrect chunking was making lance datasets use too much RAM by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2438 **Full Changelog**: lancedb/lance@v0.12.0...v0.12.1 ### [`v0.12.0`](https://togithub.com/lancedb/lance/releases/tag/v0.12.0) [Compare Source](https://togithub.com/lancedb/lance/compare/v0.11.1...v0.12.0)  #### What's Changed ##### Breaking Changes 🛠 - feat: change dataset uri to return full qualified url instead of object store path by [@eddyxu](https://togithub.com/eddyxu) in [lancedb/lance#2416 ##### New Features 🎉 - feat: new shuffler by [@BubbleCal](https://togithub.com/BubbleCal) in [lancedb/lance#2404 - feat: new index builder by [@BubbleCal](https://togithub.com/BubbleCal) in [lancedb/lance#2401 - feat: stable row id manifest changes by [@wjones127](https://togithub.com/wjones127) in [lancedb/lance#2363 - feat: once a table has been created with v1 or v2 format then it should always use that format by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2435 ##### Bug Fixes 🐛 - fix: fix file writer which was not writing page buffers in the correct order by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2413 ##### Other Changes - refactor: refactor logical decoders into "field decoders" by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2407 - refactor: rename use_experimental_writer to use_legacy_format by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2433 - refactor: minor refactor to allow I/O scheduler to be cloned in page schedulers by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2432 **Full Changelog**: lancedb/lance@v0.11.1...v0.12.0 ### [`v0.11.1`](https://togithub.com/lancedb/lance/releases/tag/v0.11.1) [Compare Source](https://togithub.com/lancedb/lance/compare/v0.11.0...v0.11.1)  #### What's Changed ##### New Features 🎉 - feat(java): support jdk8 by [@LuQQiu](https://togithub.com/LuQQiu) in [lancedb/lance#2362 - feat: support kmode with hamming distance by [@eddyxu](https://togithub.com/eddyxu) in [lancedb/lance#2366 - feat: row id index structures (experimental) by [@wjones127](https://togithub.com/wjones127) in [lancedb/lance#2303 - feat: update merge_insert to add statistics for inserted, updated, deleted rows by [@raunaks13](https://togithub.com/raunaks13) in [lancedb/lance#2357 - feat: define Flat index as a scan over VectorStorage by [@chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in [lancedb/lance#2380 - feat: add some schema utility methods to the v2 reader/writer by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2389 - feat: general compression for value page buffer by [@niyue](https://togithub.com/niyue) in [lancedb/lance#2368 - feat: make the index cache size (in bytes) available by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2381 - feat: add special uri scheme to use CloudFileReader for local fs by [@chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in [lancedb/lance#2402 - feat: add encoder utilities for pushdown by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2388 ##### Bug Fixes 🐛 - fix: concat batches before writing to avoid small IO slow down by [@chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in [lancedb/lance#2384 - fix: low recall if the num partitions is more than num rows by [@BubbleCal](https://togithub.com/BubbleCal) in [lancedb/lance#2386 - fix: f32 reduce_min for x86 by [@heiher](https://togithub.com/heiher) in [lancedb/lance#2385 - fix: fix incorrect validation logic in updater by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2408 ##### Performance Improvements 🚀 - perf: make VectorStorage and DistCalculator static to generate better code by [@BubbleCal](https://togithub.com/BubbleCal) in [lancedb/lance#2355 - perf: optimize IO path for reading manifest by [@wjones127](https://togithub.com/wjones127) in [lancedb/lance#2396 ##### Other Changes - refactor: make proto conversion fallible and not copy by [@wjones127](https://togithub.com/wjones127) in [lancedb/lance#2371 - refactor: separate take and schema evolution impls to own files by [@wjones127](https://togithub.com/wjones127) in [lancedb/lance#2372 - Revert "fix: concat batches before writing to avoid small IO slow down ([#2384](https://togithub.com/lancedb/lance/issues/2384))" by [@chebbyChefNEQ](https://togithub.com/chebbyChefNEQ) in [lancedb/lance#2387 - refactor: shuffle around v2 metadata sections to allow read-on-demand statistics by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2400 #### New Contributors - [@niyue](https://togithub.com/niyue) made their first contribution in [lancedb/lance#2368 - [@heiher](https://togithub.com/heiher) made their first contribution in [lancedb/lance#2385 **Full Changelog**: lancedb/lance@v0.11.0...v0.11.1 ### [`v0.11.0`](https://togithub.com/lancedb/lance/releases/tag/v0.11.0) [Compare Source](https://togithub.com/lancedb/lance/compare/v0.10.18...v0.11.0)  #### What's Changed ##### Breaking Changes 🛠 - feat(rust)!: use BoxedError in Error::IO by [@broccoliSpicy](https://togithub.com/broccoliSpicy) in [lancedb/lance#2329 ##### New Features 🎉 - feat: add v2 support to fragment merge / update paths by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2311 - feat: add priority to I/O scheduler by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2315 - feat: add take_rows operation to the v2 file reader's python bindings by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2331 - feat: added example for reading and writing dataset in rust by [@raunaks13](https://togithub.com/raunaks13) in [lancedb/lance#2349 - feat: new HNSW implementation by [@BubbleCal](https://togithub.com/BubbleCal) in [lancedb/lance#2353 - feat: add fragment take / fixed-size-binary support to v2 format by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2354 ##### Bug Fixes 🐛 - fix: recognize a simple expression like 'is_foo' as a scalar index query by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2356 - fix: rework list encoder to handle list-struct by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2344 - fix: minor bug fixes for v2 by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2361 ##### Documentation 📚 - docs: clearify comments in table.proto -> message DataFragment -> physical_rows by [@broccoliSpicy](https://togithub.com/broccoliSpicy) in [lancedb/lance#2346 ##### Performance Improvements 🚀 - perf: use the file metadata cache in scalar indices by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2330 ##### Other Changes - chore: remove `m_max` and `use_heuristic` params from HNSW builder by [@BubbleCal](https://togithub.com/BubbleCal) in [lancedb/lance#2336 - fix(java): fix JNI jar loader issue by [@LuQQiu](https://togithub.com/LuQQiu) in [lancedb/lance#2340 - ci: fix labeler permissions by [@wjones127](https://togithub.com/wjones127) in [lancedb/lance#2348 - fix: rework decoding to fix bugs in nested struct decoding by [@westonpace](https://togithub.com/westonpace) in [lancedb/lance#2337 #### New Contributors - [@broccoliSpicy](https://togithub.com/broccoliSpicy) made their first contribution in [lancedb/lance#2346 - [@raunaks13](https://togithub.com/raunaks13) made their first contribution in [lancedb/lance#2349 **Full Changelog**: lancedb/lance@v0.10.18...v0.11.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/spiraldb/vortex).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

chebbyChefNEQ added 2 commits May 22, 2024 19:46

Update

6baff73

[ghstack-poisoned]

Update (base update)

893ed4c

[ghstack-poisoned]

This was referenced May 22, 2024

feat: define IVFShuffler trait #2376

Closed

chore: move VectorStorage and DistCalculator to v3 module #2377

Closed

feat: add sub index trait #2378

Closed

chebbyChefNEQ added a commit that referenced this pull request May 22, 2024

feat: define Flat index as a scan over VectorStorage

43d4632

ghstack-source-id: 5d8adb1c67e07d251cc9c617dd144b7f9660242d Pull Request resolved: #2380

github-actions bot added the enhancement New feature or request label May 22, 2024

chebbyChefNEQ requested review from eddyxu, westonpace, wjones127 and BubbleCal May 22, 2024 23:47

eddyxu reviewed May 23, 2024

View reviewed changes

Update

868fd16

[ghstack-poisoned]

chebbyChefNEQ added a commit that referenced this pull request May 23, 2024

feat: define Flat index as a scan over VectorStorage

da0b532

ghstack-source-id: 64bee5f06dc4868e11e9754f73a2f7d1b68dbe41 Pull Request resolved: #2380

eddyxu approved these changes May 23, 2024

View reviewed changes

chebbyChefNEQ added 2 commits May 22, 2024 23:13

Update (base update)

7402013

[ghstack-poisoned]

Update

8a84196

[ghstack-poisoned]

chebbyChefNEQ added a commit that referenced this pull request May 23, 2024

feat: define Flat index as a scan over VectorStorage

9c65194

ghstack-source-id: dcbf9822d8e276436ed51685967fad3773be0932 Pull Request resolved: #2380

chebbyChefNEQ added 2 commits May 22, 2024 23:39

Update (base update)

29d23c9

[ghstack-poisoned]

Update

131a62b

[ghstack-poisoned]

chebbyChefNEQ added a commit that referenced this pull request May 23, 2024

feat: define Flat index as a scan over VectorStorage

bcce0ae

ghstack-source-id: 8c523fd8abb4f69352fa6998a25090e8c2ea1e6c Pull Request resolved: #2380

chebbyChefNEQ changed the base branch from gh/chebbyChefNEQ/5/base to main May 23, 2024 04:00

chebbyChefNEQ merged commit 3dd63b8 into main May 23, 2024
24 checks passed

chebbyChefNEQ deleted the gh/chebbyChefNEQ/5/head branch May 23, 2024 04:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: define Flat index as a scan over VectorStorage #2380

feat: define Flat index as a scan over VectorStorage #2380

chebbyChefNEQ commented May 22, 2024 •

edited

codecov-commenter commented May 23, 2024 •

edited

eddyxu May 23, 2024

eddyxu May 23, 2024

eddyxu left a comment

feat: define Flat index as a scan over VectorStorage #2380

feat: define Flat index as a scan over VectorStorage #2380

Conversation

chebbyChefNEQ commented May 22, 2024 • edited

codecov-commenter commented May 23, 2024 • edited

Codecov Report

eddyxu May 23, 2024

Choose a reason for hiding this comment

eddyxu May 23, 2024

Choose a reason for hiding this comment

eddyxu left a comment

Choose a reason for hiding this comment

chebbyChefNEQ commented May 22, 2024 •

edited

codecov-commenter commented May 23, 2024 •

edited