Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add additional circuit breakers #62457

Closed
lanerjo opened this issue Sep 16, 2020 · 4 comments
Closed

Add additional circuit breakers #62457

lanerjo opened this issue Sep 16, 2020 · 4 comments
Labels
:Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload >enhancement >feature Team:Core/Infra Meta label for core/infra team

Comments

@lanerjo
Copy link

lanerjo commented Sep 16, 2020

  1. breaker.search.cancel_request - This should cancel the search request across all nodes if a circuit breaker trips due to a search or aggregation.
    Currently, if one node trips a circuit breaker the request continues to run on other nodes. If using a coordinating only node, that node will still continue to receive responses from other nodes. By sending a cancel request to all other nodes that contain the request, the node could be saved in the event that many small requests would send the node over the limit. ex. 75 nodes, each sending a response in the neighborhood of 1.9 gb, circuit breaker trips on one request at 2.1gb, still another 74 nodes sending 1.9gb, which could be cancelled thereby saving the node from going unresponsive until all request finish.

  2. breaker.search.aggregation_memory.limit - Aggregation memory used (This should work better than max_buckets, as this would be more dynamic)

  3. breaker.search.calculation.memory_used - Calculations memory used (ie. unique counts)
    Ability to limit memory on a per request basis, but only for calculation being performed from that request. queries that perform unique counts on buckets of buckets on large data sets can be node killers

  4. breaker.search.request_size_limit - Total size of response from search request (should apply at the coordinating node, should also cancel the rest of the request)
    See Query DSL: Terms Filter #1 above

  5. breaker.search.request.nested_aggs_limit - Nested aggregations circuit breaker... So instead of allowing a user to nest aggregations 10+ levels, this could be tunable, and shut down before the request ever hits a shard. Admins could set a limit on the number of nested aggs.

I know this contains a list of items... I do believe that this list should be kept together as much of the work would be too closely related, and splitting them up would cause more trouble.

@lanerjo lanerjo added >enhancement needs:triage Requires assignment of a team area label labels Sep 16, 2020
@jimczi jimczi added :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload >feature and removed needs:triage Requires assignment of a team area label labels Sep 17, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/Circuit Breakers)

@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Sep 17, 2020
@soarescaique
Copy link

Can I have this one? Sounds good

@rjernst rjernst added the needs:triage Requires assignment of a team area label label Dec 3, 2020
@rjernst rjernst added team-discuss and removed needs:triage Requires assignment of a team area label labels Dec 15, 2020
@williamrandolph
Copy link
Contributor

@lanerjo

We discussed this issue in a meeting today. While the items in this list are certainly related, especially from the user's perspective, there are some distinct technical complications for different items, and the relevant expertise is spread across different groups of contributors. For example, it would be difficult to discuss, in a single thread, the architecture changes needed for task cancellation as well as the technical issues around determining how much memory an aggregation will take.

To that end, I'm going to break this issue up into separate issues.

@williamrandolph
Copy link
Contributor

We've broken this issue into separate issues for further action, so we can close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload >enhancement >feature Team:Core/Infra Meta label for core/infra team
Projects
None yet
Development

No branches or pull requests

6 participants