prune intermediate results at leaf - #6674
Conversation
prune intermediate aggregation results before sending back from leaf to root node. Otherwise a term aggregation on a high cardinality field over many splits will exceed the grpc payload.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e65106c905
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let mut merged = merged_opt.unwrap_or_default(); | ||
| // Leaf results can be merged again at the root or by a federated query. Keep the | ||
| // intermediate candidate set (`segment_size`) rather than applying final pruning. | ||
| merged.prune_intermediate_results(aggregations, PruneMode::Intermediate)?; |
There was a problem hiding this comment.
Checked the leaf merge path: IncrementalCollector::finalize merges all split responses for a leaf before calling this helper, but this line trims the merged terms buckets back to a single segment_size. That changes shard_size from per-split to per-leaf-node semantics (see docs/reference/aggregation.md:541-542), so when multiple splits are handled by the same leaf a term that was within each split's shard_size can be discarded after the leaf merge and never reach the root/federated merge, producing missing or undercounted top terms. Please preserve the per-split budget, e.g. by pruning before coalescing splits or scaling the intermediate limit by the number of merged splits.
Useful? React with 👍 / 👎.
| let serialized = | ||
| postcard::to_allocvec(&merged_opt.unwrap_or_default()).map_err(map_error)?; | ||
| let mut merged = merged_opt.unwrap_or_default(); | ||
| // Leaf results can be merged again at the root or by a federated query. Keep the |
There was a problem hiding this comment.
for root it means we increase our doc_count_error_upper_bound bound in a situation where we shouldn't
bc81132 to
092b48d
Compare
4 participants
prune intermediate aggregation results before sending back from leaf to root node.
Otherwise a term aggregation on a high cardinality field over many splits will exceed the grpc payload.