Skip to content

feat: support dynamic-bucket partial-update#317

Merged
JingsongLi merged 1 commit into
apache:mainfrom
QuakeWang:feat/partial-update-dynamic-bucket
May 14, 2026
Merged

feat: support dynamic-bucket partial-update#317
JingsongLi merged 1 commit into
apache:mainfrom
QuakeWang:feat/partial-update-dynamic-bucket

Conversation

@QuakeWang
Copy link
Copy Markdown
Contributor

Purpose

Linked issue: follow-up to #256

This PR adds basic merge-engine=partial-update support for primary-key tables using bucket=-1 dynamic bucket mode.

It extends the existing fixed-bucket partial-update support to non-cross-partition dynamic bucket tables, including partitioned tables whose primary keys include partition fields. Cross-partition dynamic bucket partial-update remains unsupported and fails fast.

Brief change log

  • Allow merge-engine=partial-update for non-cross-partition bucket=-1 PK tables.
  • Restore existing HASH index entries before assigning buckets, so updates to existing keys stay in their original buckets.
  • Keep merge-engine=partial-update with cross-partition dynamic bucket explicitly unsupported.
  • Reject partition-only primary keys, which would leave only one record per partition.
  • Add DataFusion integration tests for unpartitioned, partitioned, and existing-key dynamic bucket partial-update cases.

Tests

API and Format

This PR uses the existing dynamic bucket HASH index format and does not introduce a new storage format.

Documentation

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Copy link
Copy Markdown
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 26c3b25 into apache:main May 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants