Support batched vector index training#52
Conversation
|
Thanks for adding staged training. Before we merge this API shape, I think it is worth considering a cleaner split between training and writing, and doing that split consistently in Rust core and the Java/JNI layer. If backward compatibility is not a blocker, I would prefer not to keep training as part of
Concretely, the Rust core could expose something like: let trained = VectorIndexTrainer::new(config)?
.add_training_vectors(batch1, n1)?
.add_training_vectors(batch2, n2)?
.finish()?;
let mut writer = VectorIndexWriter::new(trained);
writer.add_vectors(&ids, &vectors, vector_count)?;
writer.write(&mut output)?;The Java/JNI API could mirror the same split: VectorIndexTraining training =
VectorIndexTrainer.create(options)
.addTrainingVectors(batch1, n1)
.addTrainingVectors(batch2, n2)
.finishTraining();
try (VectorIndexWriter writer = new VectorIndexWriter(training)) {
writer.addVectors(rowIds, vectors, vectorCount);
writer.writeIndex(output);
}A single-shot helper can still live on the trainer, for example This would make the API easier to reason about and remove the need for a writer-side state machine that mixes |
|
+1 |
What is changed
VectorIndexWriter.addTrainingVectors(...)andfinishTraining().finishTraining()completes or fails.Why
This lets Java callers feed training vectors in bounded batches instead of materializing one large
float[]for a singletrain(...)call.