feat(vindex): build indexes from streamed data splits - #707
Conversation
jerry-024
changed the title
| tokio::task::spawn_blocking(move || -> std::io::Result<()> { | ||
| let mut writer = writer; | ||
| writer.write(&mut PosWriter::new(&mut output))?; | ||
| output.shutdown() |
There was a problem hiding this comment.
Suggestion: Since a running spawn_blocking task cannot be aborted, cancelling the outer build future may still allow this worker to finish output.shutdown() after the surrounding cleanup path has been dropped. The resulting index file would not be referenced by any commit message or manifest.
If orphan index files are already covered by a separate cleanup mechanism, could we document or test that assumption here? Otherwise, it may be worth keeping a provisional output guard or another cleanup mechanism until the file is transferred into a CommitMessage.
There was a problem hiding this comment.
Good catch. I think this should be handled by orphan-file cleanup rather than adding cancellation-specific ownership logic to the index builder. A guard here would still not cover equivalent leaks caused by process/runtime shutdown, while orphan cleanup can consistently remove any unreferenced index files after the retention window. I will address the gap in the orphan cleanup path instead.
…timing * upstream/main: fix(vindex): avoid full index reads during refine (apache#708) feat(vindex): build indexes from streamed data splits (apache#707) feat: support deletion vector merge-on-read (apache#706) # Conflicts: # crates/paimon/src/table/vector_search_builder.rs # crates/paimon/src/vindex/mod.rs
3 participants
Purpose
The previous vindex builder collected every Arrow batch and then flattened all vectors into another contiguous buffer. For a 10M x 768 float32 shard, those two full-size copies alone could exceed 57 GiB before index training started.
This change builds each vindex shard from a single streamed
DataSplit, keeping builder-owned memory bounded while preserving one index file per shard.Brief change log
<index-type>.train.sample-ratioand field-level overrides; the ratio affects training only and every row is still indexed.Vec<u8>or a second local temporary file.TableCommit::abort.Tests
RUSTC_BOOTSTRAP=1 RUSTFLAGS='-Zcrate-attr=feature(stdarch_neon_f16)' cargo test -p paimon --lib vindexRUSTC_BOOTSTRAP=1 RUSTFLAGS='-Zcrate-attr=feature(stdarch_neon_f16)' cargo check -p paimon --lib --features fulltextAPI and Format
Adds the optional vindex build setting
<index-type>.train.sample-ratio, defaulting to1.0with valid range(0, 1]. There is no index storage-format change; each logical shard still produces one existing-format index file.Documentation
The SQL documentation now describes
train.sample-ratio, its valid range, and that all rows remain indexed.Notes
IVF-Flat core still retains raw vector state and creates per-list serialization copies. Chunked IVF-Flat serialization is intentionally deferred to a separate
paimon-vindex-corechange.The current FileIO async-writer wrapper cannot explicitly abort an unfinished multipart upload. Failed builds delete visible objects best-effort; object-store lifecycle rules should reclaim invisible abandoned multipart uploads.