Skip to content

feat(partial-update): support field aggregation reads #596

Description

@shyjsarah

Motivation

Paimon Rust can now read merge-engine=partial-update tables with sequence groups after #584, but it still rejects existing Java/Flink-created tables that configure:

  • fields.<field>.aggregate-function
  • fields.default-aggregate-function

This prevents Rust and DataFusion readers from opening partial-update tables that aggregate protected fields within sequence groups. Treating these tables as basic latest-non-null partial update would be incorrect, especially when an older sequence-group row must be combined in reverse order.

I searched the existing issues and did not find one covering partial-update field aggregation reads. This is a follow-up to #583 / #584.

Proposed scope

Add read-side field aggregation support for partial-update tables:

  • parse and validate per-field and default aggregate-function options;
  • reuse the aggregate functions already implemented in Paimon Rust;
  • support last_non_null_value without a sequence group, matching Java validation;
  • require other configured aggregate functions to protect fields in a sequence group;
  • apply normal aggregation when an incoming group sequence is newer or equal;
  • apply Java-compatible reversed aggregation when an incoming group sequence is older;
  • preserve sequence-group projection dependency widening;
  • fail closed with a clear error for unknown or type-incompatible aggregate functions;
  • add merge-level and end-to-end projected-read tests.

Out of scope

To keep the contribution reviewable:

  • adding Java-only aggregate-function families that are not currently implemented in Rust;
  • DELETE / UPDATE_BEFORE retraction and remove-record semantics;
  • Rust writer and CREATE/ALTER support for partial-update aggregation.

Willingness to contribute

I'm willing to submit a PR for this scoped read-side support.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions