Improve reduction of terms aggregations #61779
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Today, the terms aggregation reduces multiple aggregations at once using a map to group same buckets together. This operation can be costly since it requires to lookup every bucket in a global map with no particular order. This commit changes how term buckets are sorted by shards and partial reduces in order to be able to reduce results using a merge-sort strategy. For bwc, results are merged with the legacy code if any of the aggregations use a different sort (if it was returned by a node in prior versions). Relates elastic#51857
|
Pinging @elastic/es-analytics-geo (:Analytics/Aggregations) |
elasticmachine
added
the
Team:Analytics
nik9000
reviewed
Sep 1, 2020
server/src/main/java/org/elasticsearch/search/aggregations/InternalOrder.java
Outdated
Show resolved
Hide resolved
.../org/elasticsearch/search/aggregations/bucket/terms/GlobalOrdinalsStringTermsAggregator.java
Outdated
Show resolved
Hide resolved
| @@ -189,44 +220,173 @@ protected final void doWriteTo(StreamOutput out) throws IOException { | |||
| @Override | |||
| public abstract B getBucketByKey(String term); | |||
|
|
|||
| @Override | |||
| private static class IteratorAndCurrent<B extends InternalTerms.Bucket<B>> { | |||
There was a problem hiding this comment.
Fine for now, but I feel like we have four or five of these and maybe it'd be worth merging their implementations.
nik9000
approved these changes
Sep 1, 2020
server/src/test/java/org/elasticsearch/search/aggregations/InternalAggregationsTests.java
Outdated
Show resolved
Hide resolved
… by key and not filtered (minDocCount == 0)
|
@elasticmachine run elasticsearch-ci/packaging-sample-windows |
|
@elasticmachine run elasticsearch-ci/packaging-sample-windows |
1 similar comment
|
@elasticmachine run elasticsearch-ci/packaging-sample-windows |
jimczi
added a commit
to jimczi/elasticsearch
that referenced
this pull request
Sep 7, 2020
This commit adapts the bwc version checks added in elastic#61779 and disable bwc tests until the backport elastic#62028 is merged.
jimczi
added a commit
that referenced
this pull request
Sep 7, 2020
jimczi
added a commit
that referenced
this pull request
Sep 7, 2020
Today, the terms aggregation reduces multiple aggregations at once using a map to group same buckets together. This operation can be costly since it requires to lookup every bucket in a global map with no particular order. This commit changes how term buckets are sorted by shards and partial reduces in order to be able to reduce results using a merge-sort strategy. For bwc, results are merged with the legacy code if any of the aggregations use a different sort (if it was returned by a node in prior versions). Relates #51857
jimczi
added a commit
to jimczi/elasticsearch
that referenced
this pull request
Sep 7, 2020
jimczi
added a commit
that referenced
this pull request
Sep 7, 2020
jimczi
added a commit
to jimczi/elasticsearch
that referenced
this pull request
Nov 2, 2020
This change fixes a bug introduced in elastic#61779 that uses a compound order to compare buckets when merging. The bug is triggered when the compound order uses a primary sort ordered by key (asc or desc). This commit ensures that we always extract the primary sort when comparing keys during merging. The PR is marked as no-issue since the bug has not been released in any official version.
jimczi
added a commit
that referenced
this pull request
Nov 5, 2020
This change fixes a bug introduced in #61779 that uses a compound order to compare buckets when merging. The bug is triggered when the compound order uses a primary sort ordered by key (asc or desc). This commit ensures that we always extract the primary sort when comparing keys during merging. The PR is marked as no-issue since the bug has not been released in any official version.
jimczi
added a commit
that referenced
this pull request
Nov 5, 2020
This change fixes a bug introduced in #61779 that uses a compound order to compare buckets when merging. The bug is triggered when the compound order uses a primary sort ordered by key (asc or desc). This commit ensures that we always extract the primary sort when comparing keys during merging. The PR is marked as no-issue since the bug has not been released in any official version.
jimczi
added a commit
that referenced
this pull request
Nov 5, 2020
This change fixes a bug introduced in #61779 that uses a compound order to compare buckets when merging. The bug is triggered when the compound order uses a primary sort ordered by key (asc or desc). This commit ensures that we always extract the primary sort when comparing keys during merging. The PR is marked as no-issue since the bug has not been released in any official version.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today, the terms aggregation reduces multiple aggregations at once using a map
to group same buckets together. This operation can be costly since it requires
to lookup every bucket in a global map with no particular order.
This commit changes how term buckets are sorted by shards and partial reduces in
order to be able to reduce results using a merge-sort strategy.
For bwc, results are merged with the legacy code if any of the aggregations use
a different sort (if the shard response was created by a node in prior versions).
Relates #51857