Circuit-break on response size

[jimczi]

We have a circuit breaker for in_flight request but we don't check the response size even for APIs that can return big json output. For instance getting all _segments in a big cluster with lot of indices can kill the node responsible for the request since it needs to gather a lot of internal response as well as the final response that needs to be serialized in json. It could be beneficial to add a circuit breaker that checks the size of the transport responses when executing internal requests through the transport layer and to also control the memory usage of the response that is serialized at the rest layer. The latter is more important due to the json serialization that can be very verbose.

[elasticmachine]

Pinging @elastic/es-core-infra

[jimczi]

\cc @ywelsch @DaveCTurner

[jaymode]

The es-core-infra team discussed and we believe that this is something worth implementing as a way to add additional safeguards.

[danielmitterdorfer]

IMHO this is related to #9310. As this issue is formulated in a more generic fashion I'll close #9310 in favor of this one.

[felix-lessoer]

@elastic/es-core-infra Any updates or workarounds available?

[jasontedor]

@felix-lessoer This is not actively being worked on at this time. It’s on our list, but we are addressing more important priorities currently.

[Leaf-Lin]

I added the resiliency label because this was mentioned in the original issue #9310 and I still see this issue surfacing today.

The above issue was also mentioned from the https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html page where we stated the STATUS is still ongoing:

Add the byte size of each hit to the request circuit breaker #9310. (STATUS: ONGOING)

[vineelyalamarthy]

Are contributors welcome to look into this?

[original-brownbear]

This has become unnecessary with the introduction of chunked rest responses which turn the amount of memory needed for serialising the mentioned responses O(1). Once we're through converting all APIs that might result in a large response to chunked encoding there's no need for a circuit breaker.
relates #89838 -> closing here