Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request-level circuit breaker support on coordinating nodes #62223

Merged
merged 16 commits into from
Sep 24, 2020

Commits on Sep 10, 2020

  1. Request-level circuit breaker support on coordinating nodes

    This commit allows coordinating node to account the memory used to perform partial and final reduce of
    aggregations in the request circuit breaker. The search coordinator adds the memory that it used to save
    and reduce the results of shard aggregations in the request circuit breaker. Before any partial or final
    reduce, the memory needed to reduce the aggregations is estimated and a CircuitBreakingException} is thrown
    if exceeds the maximum memory allowed in this breaker.
    This size is estimated as roughly 1.5 times the size of the serialized aggregations that need to be reduced.
    This estimation can be completely off for some aggregations but it is corrected with the real size after
    the reduce completes.
    If the reduce is successful, we update the circuit breaker to remove the size of the source aggregations
    and replace the estimation with the serialized size of the newly reduced result.
    
    As a follow up we could trigger partial reduces based on the memory accounted in the circuit breaker instead
    of relying on a static number of shard responses. A simpler follow up that could be done in the mean time is
    to [reduce the default batch reduce size](elastic#51857) of blocking
    search request to a more sane number.
    
    Closes elastic#37182
    jimczi committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    934a3d8 View commit details
    Browse the repository at this point in the history
  2. style

    jimczi committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    2a719fa View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    74671a3 View commit details
    Browse the repository at this point in the history
  4. style

    jimczi committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    62decd3 View commit details
    Browse the repository at this point in the history
  5. fix unit test

    jimczi committed Sep 10, 2020
    Configuration menu
    Copy the full SHA
    e883199 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2020

  1. Configuration menu
    Copy the full SHA
    021f89c View commit details
    Browse the repository at this point in the history
  2. Don't serialize partial reduce eagerly

    This commit removes the serialization of partial reduce in order to speed up the merges when
    the batch reduce size is smaller than the number of shards in the request.
    The estimation of the size of partial reduce is still based on the binary size (serialized form) but we
    keep the full java object and estimate the size with a counting stream output.
    Finally this change adds a benchmark for the reduce of nested terms aggs. This benchmark was used to
    optimize the code in this PR.
    jimczi committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    603de60 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8c19f48 View commit details
    Browse the repository at this point in the history
  4. fix ut

    jimczi committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    24e708f View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4ae09cb View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    599f01d View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2020

  1. Configuration menu
    Copy the full SHA
    0a78ac8 View commit details
    Browse the repository at this point in the history
  2. address review feedback

    jimczi committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    7178dab View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8f69584 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2020

  1. Configuration menu
    Copy the full SHA
    ae90508 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    400e77a View commit details
    Browse the repository at this point in the history