Skip to content

[core] Maintain sorted primary-key index payloads#8601

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/pk-sorted-index-maintenance
Jul 13, 2026
Merged

[core] Maintain sorted primary-key index payloads#8601
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/pk-sorted-index-maintenance

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Maintain source-backed BTree and Bitmap payloads for one primary-key table bucket as compacted data files become active or inactive.

Changes

  • Restore valid rotated payload groups for each active source data file and reject incomplete, stale, malformed, or mismatched groups.
  • Track compacted source-file transitions and remove payloads whose source files are no longer active.
  • Build uncovered source files asynchronously, with bounded retries and deterministic source ordering.
  • Publish each rotated payload group atomically and delete stale, invalid, cancelled, or unpublished build output.
  • Route payload-only updates to the append increment and data-compaction updates to the compact increment.
  • Roll back source, payload, and pending-deletion state if a blocking commit is interrupted, and cancel the in-flight build.

Impact

This PR adds the bucket-local maintenance state machine only. Write-path coordination is intentionally left to the next PR. It uses the shared primary-key source policy, so complete fixed-bucket files produced by normal or Postpone Bucket compaction follow the same lifecycle.

Validation

  • mvn -pl paimon-codegen-loader -am -Pfast-build -DskipTests package
  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=BucketedSortedIndexMaintainerTest,PkSortedBucketIndexStateTest test (17 tests)
  • mvn -pl paimon-core -DskipTests validate

@JingsongLi JingsongLi marked this pull request as ready for review July 13, 2026 12:33
@JingsongLi JingsongLi merged commit be80abf into apache:master Jul 13, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants