Motivation
Paimon Rust can now read merge-engine=partial-update tables with sequence groups after #584, but it still rejects existing Java/Flink-created tables that configure:
fields.<field>.aggregate-function
fields.default-aggregate-function
This prevents Rust and DataFusion readers from opening partial-update tables that aggregate protected fields within sequence groups. Treating these tables as basic latest-non-null partial update would be incorrect, especially when an older sequence-group row must be combined in reverse order.
I searched the existing issues and did not find one covering partial-update field aggregation reads. This is a follow-up to #583 / #584.
Proposed scope
Add read-side field aggregation support for partial-update tables:
- parse and validate per-field and default aggregate-function options;
- reuse the aggregate functions already implemented in Paimon Rust;
- support
last_non_null_value without a sequence group, matching Java validation;
- require other configured aggregate functions to protect fields in a sequence group;
- apply normal aggregation when an incoming group sequence is newer or equal;
- apply Java-compatible reversed aggregation when an incoming group sequence is older;
- preserve sequence-group projection dependency widening;
- fail closed with a clear error for unknown or type-incompatible aggregate functions;
- add merge-level and end-to-end projected-read tests.
Out of scope
To keep the contribution reviewable:
- adding Java-only aggregate-function families that are not currently implemented in Rust;
- DELETE / UPDATE_BEFORE retraction and remove-record semantics;
- Rust writer and CREATE/ALTER support for partial-update aggregation.
Willingness to contribute
I'm willing to submit a PR for this scoped read-side support.
Motivation
Paimon Rust can now read
merge-engine=partial-updatetables with sequence groups after #584, but it still rejects existing Java/Flink-created tables that configure:fields.<field>.aggregate-functionfields.default-aggregate-functionThis prevents Rust and DataFusion readers from opening partial-update tables that aggregate protected fields within sequence groups. Treating these tables as basic latest-non-null partial update would be incorrect, especially when an older sequence-group row must be combined in reverse order.
I searched the existing issues and did not find one covering partial-update field aggregation reads. This is a follow-up to #583 / #584.
Proposed scope
Add read-side field aggregation support for partial-update tables:
last_non_null_valuewithout a sequence group, matching Java validation;Out of scope
To keep the contribution reviewable:
Willingness to contribute
I'm willing to submit a PR for this scoped read-side support.