Skip to content

Support dedicated vector storage writes#484

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/vector-storage-writer
Jul 8, 2026
Merged

Support dedicated vector storage writes#484
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/vector-storage-writer

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

Add writer support for dedicated vector storage so tables configured with vector.file.format write vector columns into *.vector.<format> files instead of ordinary data files. This aligns append and data-evolution partial writes with the existing reader support for .vector.* files.

Changes

  • Add vector.file.format and vector.target-file-size option accessors.
  • Validate dedicated vector storage tables require data evolution, row tracking, and a normal anchor column.
  • Split append writes so VECTOR columns are written to dedicated vector files while scalar/blob columns keep their existing layout.
  • Split data-evolution partial writes by normal versus VECTOR columns so updates also produce dedicated vector files.
  • Add focused coverage for .vector.parquet, .vector.vortex, option defaults, schema validation, and blob regression.

Testing

  • cargo test -p paimon --lib vector
  • cargo test -p paimon --lib blob_write
  • cargo test -p paimon --features vortex --lib test_vector_write_uses_dedicated_vortex_file
  • git diff --check

Notes

No migration is required. Existing inline VECTOR writes are unchanged when vector.file.format is not configured.

@JingsongLi JingsongLi force-pushed the codex/vector-storage-writer branch from c0773ed to 17efdc8 Compare July 8, 2026 10:19

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I reviewed the updated dedicated vector storage writer changes, fixed the Clippy large-enum-variant issue by boxing the dedicated writer variant, and verified the relevant paths locally.

Validated locally:

  • cargo fmt --check
  • cargo clippy -p paimon --all-targets --features vortex -- -D warnings
  • cargo test -p paimon --lib vector
  • cargo test -p paimon --features vortex --lib test_vector_write_uses_dedicated_vortex_file

The full workspace Clippy check could not be reproduced locally because this environment lacks a dynamically embeddable Python for PyO3, but the CI blocker was isolated to the paimon crate and is fixed in the latest commit.

@JingsongLi JingsongLi merged commit fb9ab68 into apache:main Jul 8, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants