Skip to content

[core][flink][spark] Support PK-vector indexes for postpone buckets#8594

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/pk-vector-postpone-bucket
Jul 13, 2026
Merged

[core][flink][spark] Support PK-vector indexes for postpone buckets#8594
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/pk-vector-postpone-bucket

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

What changed

  • Allow primary-key vector indexes in postpone-bucket mode (bucket = -2) while continuing to reject dynamic buckets.
  • Capture one snapshot for postpone planning and writer restore.
  • Make Flink and Spark background compaction process both bucket -2 rows and real-bucket Level-0 files.
  • Build and publish ANN index changes together with compacted data in one strict atomic commit.
  • Exclude postponed data from vector search until compaction and document the visibility behavior.

Why

Foreground fixed-bucket writes for postpone tables remain write-only = true. In deletion-vector mode, both real-bucket Level-0 files and bucket -2 files must be compacted before they become visible. The previous postpone compactor only consumed bucket -2, so marker-only Level-0 data could not be compacted and indexed by the background job.

Validation

  • 38 focused Core tests, including strict compact snapshot behavior.
  • Flink pinned-snapshot source test and mixed postpone/Level-0 PK-vector integration test.
  • Spark PrimaryKeyVectorSearchTest: 9 tests.
  • Spark PostponeBucketTableTest: 7 tests.
  • Spotless and Core/Flink non-fast validation.
  • git diff --check.

@JingsongLi JingsongLi marked this pull request as ready for review July 13, 2026 08:59
@JingsongLi JingsongLi changed the title Support PK-vector indexes for postpone buckets [core][flink][spark] Support PK-vector indexes for postpone buckets Jul 13, 2026
@JingsongLi JingsongLi merged commit 26f415a into apache:master Jul 13, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants