Skip to content

partial_update: support sequence-group reads#584

Merged
JingsongLi merged 2 commits into
apache:mainfrom
shyjsarah:feat/partial-update-sequence-group
Jul 22, 2026
Merged

partial_update: support sequence-group reads#584
JingsongLi merged 2 commits into
apache:mainfrom
shyjsarah:feat/partial-update-sequence-group

Conversation

@shyjsarah

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #583

Paimon Rust currently rejects existing merge-engine=partial-update tables that use fields.<sequence-fields>.sequence-group. This change adds read-side support so Rust and DataFusion can read Java/Flink-created sequence-group tables without falling back to incorrect basic latest-non-null semantics.

Brief change log

  • Parse and validate single or composite sequence fields, protected fields, multiple independent groups, missing fields, primary-key conflicts, and repeated protected fields.
  • Widen projected reads with the sequence fields required to merge projected or predicate-referenced protected fields, then remove those internal dependencies from the final projection.
  • Merge each sequence group independently using Java-compatible ascending lexicographic comparison with nulls first.
    • Skip a group when its entire sequence tuple is null.
    • Let a newer or equal sequence tuple update the group.
    • Update sequence fields atomically and allow accepted protected fields to be set to null.
  • Keep Rust CREATE/ALTER and writer paths fail-closed for sequence-group options.
  • Continue rejecting sequence-group aggregation and remove-record/delete semantics, which are outside this PR.

Tests

  • cargo test -p paimon --lib — 1797 passed, 1 ignored.
  • cargo clippy -p paimon --lib --tests -- -D warnings.
  • cargo test -p paimon-datafusion --test pk_tables — 52 passed.
  • Added coverage for independent groups, composite tuples, all-null tuple skipping, partial-null ordering, null protected-field replacement, projection dependency widening, malformed options, and end-to-end projected reads.

API and Format

No public API or storage-format changes. This only expands read compatibility for existing partial-update tables. Rust table creation and writes remain unsupported for sequence groups.

Documentation

Updated the SQL documentation with the supported read-side behavior and explicit unsupported scope.

Review notes

The comparison behavior follows Java UserDefinedSeqComparator.create(..., true): ascending field order and nulls first. Equal sequence tuples use the later row in the existing merge order. The implementation intentionally does not add ownership restrictions beyond primary-key and repeated protected-field validation used by the Java path.

@shyjsarah
shyjsarah force-pushed the feat/partial-update-sequence-group branch from d6c1f17 to c12fbd5 Compare July 22, 2026 13:22

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 5f9f184 into apache:main Jul 22, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(partial-update): support sequence-group reads

2 participants