-
Notifications
You must be signed in to change notification settings - Fork 24.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Request-level circuit breaker support on coordinating nodes #62223
Merged
jimczi
merged 16 commits into
elastic:master
from
jimczi:enhancements/reduce_aggs_circuit_breaker
Sep 24, 2020
Merged
Request-level circuit breaker support on coordinating nodes #62223
jimczi
merged 16 commits into
elastic:master
from
jimczi:enhancements/reduce_aggs_circuit_breaker
Sep 24, 2020
Commits on Sep 10, 2020
-
Request-level circuit breaker support on coordinating nodes
This commit allows coordinating node to account the memory used to perform partial and final reduce of aggregations in the request circuit breaker. The search coordinator adds the memory that it used to save and reduce the results of shard aggregations in the request circuit breaker. Before any partial or final reduce, the memory needed to reduce the aggregations is estimated and a CircuitBreakingException} is thrown if exceeds the maximum memory allowed in this breaker. This size is estimated as roughly 1.5 times the size of the serialized aggregations that need to be reduced. This estimation can be completely off for some aggregations but it is corrected with the real size after the reduce completes. If the reduce is successful, we update the circuit breaker to remove the size of the source aggregations and replace the estimation with the serialized size of the newly reduced result. As a follow up we could trigger partial reduces based on the memory accounted in the circuit breaker instead of relying on a static number of shard responses. A simpler follow up that could be done in the mean time is to [reduce the default batch reduce size](elastic#51857) of blocking search request to a more sane number. Closes elastic#37182
Configuration menu - View commit details
-
Copy full SHA for 934a3d8 - Browse repository at this point
Copy the full SHA 934a3d8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2a719fa - Browse repository at this point
Copy the full SHA 2a719faView commit details -
Configuration menu - View commit details
-
Copy full SHA for 74671a3 - Browse repository at this point
Copy the full SHA 74671a3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 62decd3 - Browse repository at this point
Copy the full SHA 62decd3View commit details -
Configuration menu - View commit details
-
Copy full SHA for e883199 - Browse repository at this point
Copy the full SHA e883199View commit details
Commits on Sep 15, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 021f89c - Browse repository at this point
Copy the full SHA 021f89cView commit details -
Don't serialize partial reduce eagerly
This commit removes the serialization of partial reduce in order to speed up the merges when the batch reduce size is smaller than the number of shards in the request. The estimation of the size of partial reduce is still based on the binary size (serialized form) but we keep the full java object and estimate the size with a counting stream output. Finally this change adds a benchmark for the reduce of nested terms aggs. This benchmark was used to optimize the code in this PR.
Configuration menu - View commit details
-
Copy full SHA for 603de60 - Browse repository at this point
Copy the full SHA 603de60View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8c19f48 - Browse repository at this point
Copy the full SHA 8c19f48View commit details -
Configuration menu - View commit details
-
Copy full SHA for 24e708f - Browse repository at this point
Copy the full SHA 24e708fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ae09cb - Browse repository at this point
Copy the full SHA 4ae09cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 599f01d - Browse repository at this point
Copy the full SHA 599f01dView commit details
Commits on Sep 17, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 0a78ac8 - Browse repository at this point
Copy the full SHA 0a78ac8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7178dab - Browse repository at this point
Copy the full SHA 7178dabView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f69584 - Browse repository at this point
Copy the full SHA 8f69584View commit details
Commits on Sep 23, 2020
-
Configuration menu - View commit details
-
Copy full SHA for ae90508 - Browse repository at this point
Copy the full SHA ae90508View commit details -
Configuration menu - View commit details
-
Copy full SHA for 400e77a - Browse repository at this point
Copy the full SHA 400e77aView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.