AggregateFnProperties

[gatesn]

pub struct AggregateFnProperties {
    pub order_sensitivity: AggregateOrderSensitivity,
    pub partial_aggregation: PartialAggregation,
    pub distinct_sensitivity: DistinctSensitivity,
}

pub enum AggregateOrderSensitivity {
    Insensitive, // planner may reorder input rows
    Sensitive,   // input order is part of the result
}

pub enum PartialAggregation {
    Unsupported,  // do not introduce partial/final aggregation
    OrderedMerge, // may split contiguous input, must merge partials in input order
    UnorderedMerge, // may split/repartition and merge partials in any order/tree
}

pub enum DistinctSensitivity {
    Sensitive,
    Insensitive, // agg(DISTINCT x) == agg(x)
}

trait AggregateFnVTable {
  ...
  fn properties(&self, options: &Self::Options, input_dtype: &DType) -> AggregateFnProperties;
}

Aggregate	Properties
count, nan_count	order-insensitive, unordered-merge
min_max	order-insensitive, unordered-merge, distinct-insensitive
is_constant	likely order-insensitive, unordered-merge
first, last, is_sorted	order-sensitive, ordered-merge
sum(bool), sum(unsigned)	order-insensitive, unordered-merge
sum(signed), sum(decimal), sum(float)	order-sensitive, partial aggregation unsupported under strict Vortex semantics
mean	inherits sum’s dtype caveats

[joseph-isaacs]

We are really asking the question AggregateOrderSensitivity is that operation commutative?

For PartialAggregation we are asking is the merge over the partial OrderedMerge -> associative and UnorderedMerge -> associative + commutative

For DistinctSensitivity is the partial idempotent

[joseph-isaacs]

we could get away with

AggregateOperationProperties  {
  commutative: bool
}

PartialMergeProperties {
  commutative: bool
  associative: bool
  idempotent: bool
}

[joseph-isaacs]

Or maybe we have a single operator struct

OperatorProperties {
    commutative: bool
    associative: bool
    idempotent: bool
}

Then this can be used for aggregations and overwritten for partial merge since aggregation op properties ==> partial merge properties.

[gatesn]

I think you want to stick to the domain-specific meaning, rather than infer it from low level properties.

e.g. is last(...) idempotent? I find I have to think much harder about it...

[joseph-isaacs]

its not, but its not clear in both cases?

[joseph-isaacs]

Okay but can we have the same meaning for both operators. agg and lifted agg (over partial).

If we use associative and commutative then we can use the same definition for aggregations and scalar_fns. I guess we could infer these but right now these are a little muddled.