Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Serialize top-level pipeline aggs as part of InternalAggregations #40177

Merged
merged 5 commits into from
Mar 19, 2019
Merged

Serialize top-level pipeline aggs as part of InternalAggregations #40177

merged 5 commits into from
Mar 19, 2019

Commits on Mar 18, 2019

  1. Serialize top-level pipeline aggs as part of InternalAggregations 

    We currently convert pipeline aggregators to their corresponding
InternalAggregation instance as part of the final reduction phase.
They arrive to the coordinating node as part of QuerySearchResult
objects fom the shards and, despite we may incrementally reduce
aggs (hence we may have some non-final reduce and the final
one later) all the reduction phases happen on the same node.

With CCS minimizing roundtrips though, each cluster performs its
own non-final reduction, and then serializes the results back to
the CCS coordinating node which will perform the final coordination.
This breaks the assumptions made up until now around reductions
happening all on the same node.

With elastic#40101 we have made sure that top-level pipeline aggs are not
reduced as part of the non-final reduction. The next step is to make
sure that they don't get lost, meaning that each coordinating node
needs to send them back to the CCS coordinating node as part of
the top-level `InternalAggregations` object.

Closes elastic#40059
    @javanna
    javanna committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    295ee9f View commit details
    Browse the repository at this point in the history

  2. line length

    @javanna
    javanna committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    876c6ca View commit details
    Browse the repository at this point in the history

  3. license header

    @javanna
    javanna committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    f1fe3a8 View commit details
    Browse the repository at this point in the history

  4. address comments

    @javanna
    javanna committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    88c4728 View commit details
    Browse the repository at this point in the history

  5. address test failures

    @javanna
    javanna committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    5cbfedc View commit details
    Browse the repository at this point in the history