[Documentation Request] Document how to stream results of large search queries

[cyrille-leclerc]

Description

Search queries can return a large volume of data and c.e.c.e.core.search.HitsMetadata#hits() is a List rather than an a kind of Iterator suggesting that the results are all loaded in memory rather than being streamed similarly to the java.sql.ResultSet API.

Could we have documentation on best practices to stream large search responses?

The Jenkins OpenTelemetry Plugin has implemented an abstraction of Iterator that progressively retrieves small chunks of and Elasticsearch query using the Point in Time API (code here) but it may not be the most efficient solution.

[swallez]

A SQL ResultSet is a sequence of independent messages sent over the wire, and lazy iteration is possible there. A JSON response is a tree where properties can come in arbitrary order. So we have to read the entire tree to be able to return a consistent response, and so a search response's hits are all loaded in memory, and this is really a List.

The approach taken in the OpenTelemetry plugin is the right one, by using the Point in Time API to split a large result set into smaller chunks and provide an Iterator on top of it.

This approach is what we plan to use to provide a PointInTimeHelper. Bear with us, it's on our roadmap, and documentation will come with this helper. However we will not write documentation explaining how to use the Point In Time API specifically from the Java API client, as it would not add much value compared to the corresponding Elasticsearch API doc and would have to trashed once the PointInTimeHelper is there.

[cyrille-leclerc]

Thanks @swallez,

However we will not write documentation explaining how to use the Point In Time API specifically from the Java API client, as it would not add much value compared to the corresponding Elasticsearch API doc and would have to trashed once the PointInTimeHelper is there.

I guess it could be helpful in the Javadocs of the Point in Time API to have links to the Elasticsearch HTTP API docs then.

[swallez]

I guess it could be helpful in the Javadocs of the Point in Time API to have links to the Elasticsearch HTTP API docs then.

You're absolutely right, and it's actually in the javadoc 😉

[cyrille-leclerc]

Thanks.
I guess few other javadocs could also help:

• co.elastic.clients.elasticsearch.core.SearchRequest.Builder#pit(java.util.function.Function<co.elastic.clients.elasticsearch.core.search.PointInTimeReference.Builder,co.elastic.clients.util.ObjectBuilder<co.elastic.clients.elasticsearch.core.search.PointInTimeReference>>) : the search request builder is an API you manipulate even if you don't know what Point In Times are
• co.elastic.clients.elasticsearch.core.SearchResponse#pitId: ditto

Cyrille

[swallez]

Documentation on SearchRequest.pit has been added to the specification and is now in the javadoc.