feat: new index builder

[BubbleCal]

No description provided.

[chebbyChefNEQ]

why is this unimplemented?

Another nit would be, maybe we should rename this struct to FloatVectorStorage?

[BubbleCal]

still working on it, going to change the field to FixedSizeList so it can store any types of vector, and rename it to FlatVectorStorage

[chebbyChefNEQ]

we probably want a way to control concurrency behavior here. For some index types it make sense to build one by one, for other index types we may want concurrent builds

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 53.87701% with 345 lines in your changes are missing coverage. Please review.

Project coverage is 79.52%. Comparing base (32cd740) to head (927376e).

Files	Patch %	Lines
rust/lance-index/src/vector/pq/storage.rs	1.35%	73 Missing ⚠️
rust/lance/src/index/vector/builder.rs	75.19%	20 Missing and 45 partials ⚠️
rust/lance-index/src/vector/flat/index.rs	24.70%	64 Missing ⚠️
rust/lance-index/src/vector/sq/storage.rs	19.48%	60 Missing and 2 partials ⚠️
rust/lance-index/src/vector/quantizer.rs	4.34%	22 Missing ⚠️
rust/lance-index/src/vector/flat/storage.rs	79.80%	19 Missing and 2 partials ⚠️
rust/lance-index/src/vector/utils.rs	68.75%	10 Missing ⚠️
rust/lance/src/index/vector/utils.rs	52.94%	7 Missing and 1 partial ⚠️
rust/lance/src/index/vector/ivf.rs	45.45%	6 Missing ⚠️
rust/lance-index/src/vector/ivf.rs	82.14%	4 Missing and 1 partial ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2401      +/-   ##
==========================================
- Coverage   79.59%   79.52%   -0.08%     
==========================================
  Files         203      205       +2     
  Lines       55634    56242     +608     
  Branches    55634    56242     +608     
==========================================
+ Hits        44284    44724     +440     
- Misses       8790     8877      +87     
- Partials     2560     2641      +81     

Flag	Coverage Δ
unittests	79.52% <53.87%> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chebbyChefNEQ]

Let's create a few ticket to make sure we clean things up?

• Save and load IVF mode
• Use setter/build pattern for IvfIndexBuilder so we can have more succinct APIs
• Add more tests
  • e2e -- build index + search
• Documentation describing how different traits and struct works and how they fit together.