Prevent storing huge async search responses

[mayya-sharipova]

We saw an issue when storing a huge async search response lead to OOM.

This issue is about finding ways to prevent this. We've discussed possible ways to prevent this from happening:

1. Introduce an additional circuit breaker on a coordinating node for fetch phase. In Early detection of circuit breaker exception in the coordinating node #67431, we've introduced a circuit breaker for aggs for a coordinating node, but we still don't have a circuit breaker for top docs on the fetch phase. This circuit breaker will trip when memory used for collected top docs exceeds a Request circuit breaker size.

2. Introduce a circuit breaker on indexing huge docs.

3. Store async search response as an object with enabled=false instead of storing in a stored field. When we store async response in a stored field, we first encode it with base 64 encoding which needs additional memory. If we avoid base 64 encoding, we may need much less memory. But here a question is how to preserve the es version when this response was recorded?

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[jtibshirani]

From digging into the errors we've seen, there are a couple different sources of problems:

• We serialize the in-memory search response to a byte array, then base64 encode this to a string. First, we can OOM while allocating the byte array. Second, if the length of the encoded response exceeds ~2GB, then we'll also OOM. We could try to catch these cases to avoid OOMing and instead throw a normal exception (with a clear error message)?
• We use an update request to write the response to the async search index. Update requests must parse the document source to a map to merge it into the existing document. This can use ~3x the memory of the original request which has caused OOMs. Is there any way to rework the async search code to use an index request instead of update, to avoid the memory overhead?
• Encoding and writing the search response can consume extra memory -- can we account for this additional memory usage in circuit breaking?

These are short-term fixes that would immediately help with the problems we're seeing. In parallel we may want to step back and generally think through how to be robust to large search responses (we've discussed ideas like circuit breaking the fetch phase, truncating field values, limiting document size at index time...). I don't have a clear idea for this yet.

[jtibshirani]

After more thought and discussion with others, I think it'd make the most sense to start with a soft limit on the size of the async search response we attempt to store and return. If coordinating node detects the serialized response would be larger than the limit, we'd throw an error early to avoid memory problems. The limit could be configured through a cluster setting.

The thinking is that even if we addressed the points in the comment above, we could see problems when indexing the document, since to create stored fields, we copy the document source into a BytesRef (which is backed by a byte array). Also having very large documents in the .async_search index can lead to high memory usage down the line, for example while merging stored fields.

Summary of proposal:

• Introduce a soft limit on the size of the async search response. We'd need to choose the limit carefully to avoid blocking too many searches that were previously possible.
• As a complement, look into "low hanging fruit" to reduce memory usage on coordinating node and add circuit breaking. (For example, it may be possible to reduce overhead during encoding from 3x to ~2x?)

We could also look into improvements that help make the search response smaller, so that we reject as few searches as possible. We've discussed some ideas like truncating large fields (#72453), and compressing the response before storing it.

[jtibshirani]

Here are the fixes we made (thanks to @mayya-sharipova and @dnhatn!) In summary, 7.14 will bring stability improvements, but the issue won't be fully addressed until 7.15.

7.14 Check circuit breaker when encoding and decoding response, and avoid unnecessarily copying data.

• Integrate circuit breaker in AsyncTaskIndexService #73862
• Check memory usage before decoding async response #74594
• Write async response directly to XContent to reduce memory usage #73707

7.15 Add a soft limit on the size of the response we will attempt to store. Compress the response to reduce size of documents and help avoid hitting the limit.

• Set max allowed size for stored async response #74455
• Compress async search responses before storing #74766

Follow-ups planned for 7.15:

• In 7.x the soft limit is disabled by default. We're discussing whether we can enable it by default to a large number like we do on master (10MB).
• The limit is currently disabled for EQL responses, should we enable it as well?

[mayya-sharipova]

@jtibshirani Thanks, that's a great summary!

@julie I've doubled checked our action on the failure of createResponse, and besides recording a failure in the logs, we also return a failure response to a user, so we don't need to do anything extra here.

[jtibshirani]

I added the two follow-ups to the async search meta issue (#88658). Closing this in favor of that issue.