Seperate quantizer training from index build - #235
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR separates the quantizer training phase from the main heap scan in the index build process and ensures the index-creation progress is updated correctly for plain storage.
- Removed
start_training/add_sample/finish_trainingmethods from theStoragetrait and its implementations. - Introduced a standalone
train_quantizerfunction (withBqTrainState) to handle quantizer training before graph building. - Updated
ambuildanddo_heap_scanto accept and forwardWriteStats, reflecting the split between training and building.
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/access_method/storage.rs | Dropped unused training methods from the Storage trait. |
| src/access_method/sbq/storage.rs | Switched SBQ storage to load a pretrained quantizer; removed inline training methods. |
| src/access_method/plain/storage.rs | Removed no-op training stubs to align with trait cleanup. |
| src/access_method/graph/mod.rs | Deleted get_meta_page_mut accessor as it’s no longer used. |
| src/access_method/build.rs | Added train_quantizer, restructured ambuild for two-phase building, and updated do_heap_scan. |
Comments suppressed due to low confidence (3)
pgvectorscale/src/access_method/build.rs:32
- [nitpick] The struct name
BqTrainStateis inconsistent with theSbqprefix used elsewhere; consider renaming it toSbqTrainStatefor clarity and consistency.
struct BqTrainState<'a, 'b> {
pgvectorscale/src/access_method/build.rs:277
- [nitpick] Consider adding a doc comment to
train_quantizerto explain its purpose, parameters, return value, and when it should be used.
fn train_quantizer(
pgvectorscale/src/access_method/build.rs:277
- It would be beneficial to add unit or integration tests covering
train_quantizerto ensure the quantizer training logic and metadata storage behave as expected.
fn train_quantizer(
syvb
force-pushed
the
sv/quantize-seperately
branch
from
June 6, 2025 16:52
ec1f506 to
eb8521f
Compare
syvb
force-pushed
the
sv/quantize-seperately
branch
from
June 6, 2025 16:53
eb8521f to
2876d29
Compare
tjgreen42
previously approved these changes
Jun 6, 2025
tjgreen42
left a comment
Collaborator
There was a problem hiding this comment.
Looks good to me, modulo minor comment about naming!
syvb
force-pushed
the
sv/quantize-seperately
branch
from
June 10, 2025 16:55
cdbf81b to
41384f9
Compare
tjgreen42
approved these changes
Jun 10, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This splits out quantizer training from
do_heap_scan. This doesn't change any of the behaviour of index-building, but is going to be useful in a future PR that adds support for parallel indexing.This PR also makes it so the creation progress is set properly when building a plain (non-quantized) index.