Pipeline aggregations: Ability to perform computations on aggregations

[colings86]

Adds a new type of aggregation called 'reducers' which act on the output of aggregations and compute extra information that they add to the aggregation tree. Reducers look much like any other aggregation in the request but have a buckets_path parameter which references the aggregation(s) to use.

Internally there are two types of reducer; the first is given the output of its parent aggregation and computes new aggregations to add to the buckets of its parent, and the second (a specialisation of the first) is given a sibling aggregation and outputs an aggregation to be a sibling at the same level as that aggregation.

This PR includes the framework for the reducers, the derivative reducer (#9293), the moving average reducer(#10002) and the maximum bucket reducer(#10000). These reducer implementations are not all yet fully complete.

Known work left to do (these points will be done once this PR is merged into the master branch):

• Add x-axis normalisation to the derivative reducer
• Add lots more JUnit tests for all reducers

Contributes to #9876, #10002, #9293, and #10000

[jpountz]

Should it be aggPathAsList.get(aggPathAsList.size() - 1).equals("_count") ?

[colings86]

Not sure it matters? If the whole path is _count then it will work as is, and you can't get deeper buckets if there is a doc count of 0 in the top level anyway?

[jpountz]

I'm concerned this might hide actual issues?

[colings86]

Should we add a debug log here then? it still needs to return null as a bucket may not have an entry for a particular aggregation name

[jpountz]

I like the fact that it's not too invasive and makes efforts to not have side-effects on other aggs by copying the agg trees instead of modifying them in-place.

Most of my comments are cosmetics, in particular there are lots of indentation issues in aggregator constructor and anonymous LeafCollector definitions.

Regarding documentation, I think we need to move general informations to the main page about aggs/reducers instead of having it scattered across individual reducers/aggs.

[jpountz]

I like the fact that it's not too invasive and makes efforts to not have side-effects on other aggs by copying the agg trees instead of modifying them in-place.

Most of my comments are cosmetics, in particular there are lots of indentation issues in aggregator constructor and anonymous LeafCollector definitions.

Regarding documentation, I think we need to move general informations to the main page about aggs/reducers instead of having it scattered across individual reducers/aggs.

[colings86]

@jpountz I pushed an update and replied to some of your comments. Any chance you could have another look?

[colings86]

@jpountz I pushed an update and replied to some of your comments. Any chance you could have another look?

[jpountz]

It looks to me like there are still some open TODOs about documentation but other than that LGTM. I pushed commits for some indentation issues I found.

[jpountz]

It looks to me like there are still some open TODOs about documentation but other than that LGTM. I pushed commits for some indentation issues I found.

[polyfractal]

I'm working on the larger doc TODOs (e.g. centralize docs about _count, buckets_path and path syntax, gap_policy, etc). I don't think it should block merging this though, especially since I'm restructuring part of the aggs docs as a whole and want some eyeballs...don't want to hold up this PR.

[polyfractal]

I'm working on the larger doc TODOs (e.g. centralize docs about _count, buckets_path and path syntax, gap_policy, etc). I don't think it should block merging this though, especially since I'm restructuring part of the aggs docs as a whole and want some eyeballs...don't want to hold up this PR.