From 986a5741b97ed57cf5afc6f899a151c58b16bfe0 Mon Sep 17 00:00:00 2001 From: Bogdan Pintea Date: Fri, 28 Mar 2025 13:37:15 +0100 Subject: [PATCH 1/3] Improve docs on LIMIT This adds a few extra details around how ESQL processes input docs and how it limits output results. --- .../query-languages/esql/limitations.md | 38 ++++++++++++++++++- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/docs/reference/query-languages/esql/limitations.md b/docs/reference/query-languages/esql/limitations.md index b030c089f2523..19ed7596a35fd 100644 --- a/docs/reference/query-languages/esql/limitations.md +++ b/docs/reference/query-languages/esql/limitations.md @@ -10,20 +10,54 @@ mapped_pages: ## Result set size limit [esql-max-rows] -By default, an {{esql}} query returns up to 1000 rows. You can increase the number of rows up to 10,000 using the [`LIMIT`](/reference/query-languages/esql/esql-commands.md#esql-limit) command. Queries do not return more than 10,000 rows, regardless of the `LIMIT` command’s value. +By default, an {{esql}} query returns up to 1,000 rows. You can increase the number of rows up to 10,000 using the [`LIMIT`](/reference/query-languages/esql/esql-commands.md#esql-limit) command. -This limit only applies to the number of rows that are retrieved by the query. Queries and aggregations run on the full data set. +For instance, +```esql +FROM index | WHERE field = "value" +``` +is equivalent to: +```esql +FROM index | WHERE field = "value" | LIMIT 1000 +``` + +Queries do not return more than 10,000 rows, regardless of the `LIMIT` command’s value. This is a configurable upper limit. To overcome this limitation: * Reduce the result set size by modifying the query to only return relevant data. Use [`WHERE`](/reference/query-languages/esql/esql-commands.md#esql-where) to select a smaller subset of the data. * Shift any post-query processing to the query itself. You can use the {{esql}} [`STATS`](/reference/query-languages/esql/esql-commands.md#esql-stats-by) command to aggregate data in the query. +The upper limit only applies to the number of rows that are output by the query, not to the number of documents it processes: the query runs on the full data set. + +Consider the following two queries: +```esql +FROM index | WHERE field0 == "value" | LIMIT 20000 +``` +and +```esql +FROM index | STATS AVG(field1) BY field2 | LIMIT 20000 +``` + +In both cases, the filtering by `field0` in the first query or the grouping by `field2` in the second is applied over all the documents present in the `index`, irrespective of their number or indexes size. However, both queries will return at most 10,000 rows, even if there were more rows available to return. + The default and maximum limits can be changed using these dynamic cluster settings: * `esql.query.result_truncation_default_size` * `esql.query.result_truncation_max_size` +However, doing so involves trade-offs. A larger result-set involves a higher memory pressure and increased processing times; the internode traffic within and across clusters can also increase. + +These limitations are akin those enforced by the [search API](/reference/elasticsearch/rest-apis/paginate-search-results.html#from-and-size-pagination). + +| Functionality | Search | {{esql}} | +|----------------------------------|-------------------------|-------------------------------------------| +| Results returned by default | 10 | 1.000 | +| Default upper limit | 10,000 | 10,000 | +| Specify number of results | `size` | `LIMIT` | +| Change default number of results | n/a | esql.query.result_truncation_default_size | +| Change default upper limit | index-max-result-window | esql.query.result_truncation_max_size | + ## Field types [esql-supported-types] From 6e2821ca7cfa8d4258e6dea8f7a010d30c94336c Mon Sep 17 00:00:00 2001 From: Bogdan Pintea Date: Fri, 28 Mar 2025 19:17:39 +0100 Subject: [PATCH 2/3] Update docs/reference/query-languages/esql/limitations.md Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> --- docs/reference/query-languages/esql/limitations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/query-languages/esql/limitations.md b/docs/reference/query-languages/esql/limitations.md index 19ed7596a35fd..35b58a601d4bf 100644 --- a/docs/reference/query-languages/esql/limitations.md +++ b/docs/reference/query-languages/esql/limitations.md @@ -48,7 +48,7 @@ The default and maximum limits can be changed using these dynamic cluster settin However, doing so involves trade-offs. A larger result-set involves a higher memory pressure and increased processing times; the internode traffic within and across clusters can also increase. -These limitations are akin those enforced by the [search API](/reference/elasticsearch/rest-apis/paginate-search-results.html#from-and-size-pagination). +These limitations are similar to those enforced by the [search API for pagination](/reference/elasticsearch/rest-apis/paginate-search-results.md#from-and-size-pagination). | Functionality | Search | {{esql}} | |----------------------------------|-------------------------|-------------------------------------------| From fdcd4a0f705ef61e85fc2414e1531dac4311fde7 Mon Sep 17 00:00:00 2001 From: Bogdan Pintea Date: Fri, 28 Mar 2025 19:36:47 +0100 Subject: [PATCH 3/3] slug fix --- docs/reference/query-languages/esql/limitations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/query-languages/esql/limitations.md b/docs/reference/query-languages/esql/limitations.md index 35b58a601d4bf..4aca7feb4c19e 100644 --- a/docs/reference/query-languages/esql/limitations.md +++ b/docs/reference/query-languages/esql/limitations.md @@ -48,7 +48,7 @@ The default and maximum limits can be changed using these dynamic cluster settin However, doing so involves trade-offs. A larger result-set involves a higher memory pressure and increased processing times; the internode traffic within and across clusters can also increase. -These limitations are similar to those enforced by the [search API for pagination](/reference/elasticsearch/rest-apis/paginate-search-results.md#from-and-size-pagination). +These limitations are similar to those enforced by the [search API for pagination](/reference/elasticsearch/rest-apis/paginate-search-results.md#paginate-search-results). | Functionality | Search | {{esql}} | |----------------------------------|-------------------------|-------------------------------------------|