add memory limit for aggregations #1942
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PSeitz
force-pushed
the
agg_memory_limit
branch
2 times, most recently
from
ebabd4d
to
1d305c5
Compare
PSeitz
force-pushed
the
agg_memory_limit
branch
from
1d305c5
to
8b49378
Compare
fulmicoton
reviewed
Mar 15, 2023
fulmicoton
reviewed
Mar 15, 2023
| fn memory_consumption(&self) -> usize; | ||
| } | ||
|
|
||
| impl<K, V, S> MemoryConsumption for HashMap<K, V, S> { |
There was a problem hiding this comment.
Suggested change
| impl<K, V, S> MemoryConsumption for HashMap<K, V, S> { | |
| impl<K: Copy, V: Copy, S> MemoryConsumption for HashMap<K, V, S> { |
fulmicoton
reviewed
Mar 15, 2023
| #[error("Date histogram parse error: {0:?}")] | ||
| DateHistogramParseError(#[from] DateHistogramParseError), | ||
| /// Memory limit exceeded | ||
| #[error( | ||
| "Aborting aggregation because memory limit was exceeded. Limit: {limit:?}, Current: \ |
There was a problem hiding this comment.
that would be nice to display a value expressed in MB, with a unit.
There was a problem hiding this comment.
I changed the existing ByteCount to cover that
fulmicoton
reviewed
Mar 15, 2023
| @@ -55,7 +55,7 @@ impl fmt::Debug for DataCorruption { | |||
| #[derive(Debug, Clone, Error)] | |||
| pub enum TantivyError { | |||
| /// Error when handling aggregations. | |||
| #[error("AggregationError {0:?}")] | |||
| #[error(transparent)] | |||
There was a problem hiding this comment.
ah cool ! I did not know about transparent
fulmicoton
reviewed
Mar 15, 2023
fulmicoton
approved these changes
Mar 15, 2023
introduce AggregationLimits to set memory consumption limit and bucket limits memory limit is checked during aggregation, bucket limit is checked before returning the aggregation request.
Co-authored-by: Paul Masurel <paul@quickwit.io>
PSeitz
force-pushed
the
agg_memory_limit
branch
from
81c523a
to
384592f
Compare
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## main #1942 +/- ##
========================================
Coverage 94.48% 94.48%
========================================
Files 309 311 +2
Lines 56999 57207 +208
========================================
+ Hits 53855 54054 +199
- Misses 3144 3153 +9
... and 4 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
This was referenced Mar 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
introduce AggregationLimits to set memory consumption limit and bucket limits
memory limit is checked during aggregation, bucket limit is checked before returning the aggregation request.
TODO: add check for term aggregation