Include hits in request circuit breaker #9310

Open
clintongormley opened this Issue Jan 15, 2015 · 4 comments

Projects

None yet

6 participants

@clintongormley
Member

The bytes from the _source field and highlighting returned during the FETCH phase should be included in the request circuit breaker.

@jpountz
Contributor
jpountz commented Jan 30, 2015

I think there are several things to think about:

  • do we want to do it on a per-request basis or globally (globally introduces serious complexity as we need to decrease the counter when memory is not used anymore, and use locks to ensure correctness of the counter)
  • we do not know how much memory stored fields use before we load them, so we could not really protect against super large documents
  • when serializing the response, we take the response object, serialize it, and only then start sending data over the network. So the actual total amount of memory that is used to handle the fetch phase would actually be close to twice the data that comes from stored fields + highlights.

Overall I am a bit concerned such a feature would be very hard to implement correctly while addressing #9311 would already solve most problems.

@kimchy
Member
kimchy commented Jan 30, 2015

@jpountz I think this should use our request circuit breaker mechanism, that already works well in the context of aggs (in terms of releasing it at the correct search execution phase) and is "global". My thought was simply to add to the circuit breaker once each document is loaded. This will also help to take into account a more bigger notion of memory usage, basically anything the request circuit breaker encapsulates.

@s1monw s1monw added v1.6.0 and removed v1.6.0 v1.5.0 labels Mar 17, 2015
@clintongormley clintongormley added v2.0.0 and removed v1.6.0 labels May 29, 2015
@clintongormley clintongormley added v2.2.0 and removed v2.1.0 labels Nov 20, 2015
@spinscale spinscale added v2.3.0 and removed v2.2.0 labels Dec 23, 2015
@clintongormley clintongormley added v2.4.0 and removed v2.3.0 labels Mar 16, 2016
@clintongormley clintongormley added v2.4.1 and removed v2.4.0 labels Aug 24, 2016
@clintongormley
Member

@danielmitterdorfer does the inflight request circuit breaker handle this situation already?

@danielmitterdorfer
Member

@clintongormley No, the inflight-request circuit breaker tracks only bytes for request processing (see TcpTransport#handleRequest()). Bytes allocated for responses are not counted.

@clintongormley clintongormley added v2.4.2 and removed v2.4.1 labels Sep 23, 2016
@clintongormley clintongormley removed the v2.4.2 label Nov 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment