feat(cli): create index --train-pq-codebook trains codebooks at creation (#920 sub-item 3) - #939
Merged
Merged
Conversation
…ion (refs #920) Add a --train-pq-codebook <jsonl> flag to `laurus create index` that trains the shared PQ codebook for every eligible HNSW field (ProductQuantization + pq_codebook_path, field-name-sorted) immediately after creation, so the very first commit can already encode against it — removing the create → train → ingest ordering the #918 failure policy otherwise leaves the user to manage manually. Validation (JSONL exists, at least one eligible field) runs before anything is created, so a bad invocation never leaves a half-initialized index. The effective schema follows init_index's recovery rule (an existing schema.toml without store/ wins over the argument/wizard schema). The JSONL collection loop is extracted from `train pq-codebook` into a shared helper; sub-item 2 (FastScan) of #920 remains open.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--from-indexmerged as PR feat(cli): train pq-codebook --from-index samples committed vectors (#920 sub-item 1) #938; sub-item 2 FastScan remains open). Refs feat(vector/index): PQ shared-codebook follow-ups (--from-index, FastScan, create convenience flag) (#631 PR-4) #920 — does not close it.--train-pq-codebook <jsonl>tolaurus create index: shared PQ codebooks are trained as part of index creation, so the very first commit can already encode against them. This removes the create →train pq-codebook→ ingest ordering hazard that the feat(vector,cli,server): schema-level pq_codebook_path + laurus train pq-codebook (#631 PR-2) #918 failure policy (hard-error at commit, no silent fallback) otherwise leaves the user to manage manually.lauruslibrary crate is untouched — changes are confined tolaurus-cli+ docs.Changes
train.rs: JSONL collection loop extracted intopub(crate) collect_vectors_from_jsonl(behavior-preserving; the 7 existing train tests are the regression gate).cli.rs/main.rs:CreateResource::Indexgainstrain_pq_codebook: Option<PathBuf>.create.rs::run_index:init_index's recovery rule (an existingschema.tomlwithoutstore/wins over the argument/wizard schema); all paths now go throughcreate_index_from_schema.Hnsw + ProductQuantization + pq_codebook_path, sorted by field name sinceSchema::fieldsis a HashMap) withoutput=None, so the persisted schema and trained file agree by construction.laurus-cli/commands.mdcreate-index section (usage, argument row, one-step example);vector_indexing.mdshared-codebook section.Tests
4 new in
create.rs(CLI total 17):create_with_train_flag_makes_first_commit_succeed— the direct hazard-resolution regression (.pqcbexists, reopened first commit succeeds).create_without_train_flag_leaves_first_commit_failing— control pinning the hazard itself: without the flag the same schema's first commit fails with thepq_codebook_pathhard-error. Together these form a RED/GREEN pair attributing the fix causally to the flag.create_with_train_flag_rejects_schema_without_pq_field— bails and nothing is created.create_with_train_flag_trains_every_pq_field— multiple PQ fields all trained.Verification
cargo fmt --check/cargo clippy --all-targets -- -D warnings: clean on stable and 1.97.0.cargo test -p laurus-cli --bin laurus: 17 passed (+4).cargo test -p laurus --lib: 1273 passed (library crate untouched).