Skip to content

feat(aggregate): add Java-compatible field aggregators - #183

Open
liangjie3138 wants to merge 3 commits into
apache:mainfrom
liangjie3138:feat/java-compatible-aggregators
Open

feat(aggregate): add Java-compatible field aggregators#183
liangjie3138 wants to merge 3 commits into
apache:mainfrom
liangjie3138:feat/java-compatible-aggregators

Conversation

@liangjie3138

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close alibaba/paimon-cpp#457

Port the six practical field aggregators missing on the C++ side — collect,
merge_map, nested_update, rbm64, hll_sketch, theta_sketch .

Also adds GenericArray / GenericMap, the in-memory InternalArray /
InternalMap implementations these aggregators need to build ARRAY and MAP
results.

Tests

18 new unit test cases in paimon-core-test, covering each aggregator's
aggregate / retract / null / type-validation paths.

Three pin behaviour that is easy to "fix" incorrectly, and each was checked by
reverting the corresponding production change and confirming it fails:

  • BinaryAggMergeFunctionTest.OwnedAccumulatorSurvivesNullInput — drives a real
    AggregateMergeFunction so the use-after-free is exercised through the
    production path.
  • FieldNestedUpdateAggTest.CountLimitCountsNullElementsOfAccumulator — count
    limit is measured against the raw element count, matching Java.
  • FieldIgnoreRetractAggTest.ReversedAggBypassesWrappedOverride — the wrapper
    does not preserve the wrapped aggregator's reversed-aggregation override, also
    matching Java.

No integration tests added.

API and Format

Additive only, no signatures changed:

  • include/paimon/defs.h: four new Options constants — NESTED_KEY,
    NESTED_KEY_NULL_STRATEGY, NESTED_SEQUENCE_FIELD, COUNT_LIMIT.
  • CoreOptions: a NestedKeyNullStrategy enum plus accessors.
  • FieldAggregator: new virtuals default to the existing Agg / AggReversed.

New dependency: Apache DataSketches 5.2.0 (Apache-2.0), resolved through
resolve_dependency() like the other bundled dependencies.

Documentation

None.

Generative AI tooling

Generated-by: Claude Code 2.1.220 (Claude Opus 5)

@liangjie3138
liangjie3138 force-pushed the feat/java-compatible-aggregators branch from d62f43c to b9d5c45 Compare August 5, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add practical field aggregators available in Java Paimon

1 participant